recently while working on MySQL cluster 7.3.5, we came across this error
Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
The management nodes would start up but
the data nodes would complete phase 1 & shutdown, some times at phase 2 and some times at phase 5.
The error one would see in the trace file would be as follows:
Status: Temporary error, restart node Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug) Error: 2341 Error data: CREATE_TABLE_REF Error object: NDBCNTR (Line: 2493) 0x00000003 Program: ndbd
How did we resolve it ?
First, few of our data nodes had an incorrect connect-string
Second, we had pre-assigned NodeGroups for the data nodes in the config.ini, and had started our node groups with serial number 1.
[ndbd]
NodeGroupt=1
HostName=10.95.139.92
[ndbd]
NodeGroupt=2
HostName=10.95.139.92
Apparently, NDB likes to start up with serial number 0 onwards, so we changed to
[ndbd]
NodeGroupt= 0
HostName=10.95.139.92
[ndbd]
NodeGroupt= 1
HostName=10.95.139.92
and walaa our cluster started up.
Thanks to Frank, for giving the work around (here).