NDB – Forced node shutdown completed. Caused by error 2341: ‘Internal program error (failed ndbrequire)

recently while working on MySQL cluster 7.3.5, we came across this error

Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

The management nodes would start up but

the data nodes would complete phase 1 & shutdown, some times at phase 2 and some times at phase 5.

The error one would see in the trace file would be as follows:

Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: CREATE_TABLE_REF
Error object: NDBCNTR (Line: 2493) 0x00000003
Program: ndbd

How did we resolve it ?

First, few of our data nodes had an incorrect connect-string

Second, we had pre-assigned NodeGroups for the data nodes in the config.ini, and had started our node groups with serial number 1.

[ndbd]

NodeGroupt=1

HostName=10.95.139.92

[ndbd]

NodeGroupt=2

HostName=10.95.139.92

Apparently, NDB likes to start up with serial number 0 onwards, so we changed  to

[ndbd]

NodeGroupt= 0

HostName=10.95.139.92

[ndbd]

NodeGroupt= 1

HostName=10.95.139.92

and walaa our cluster started up.

Thanks to Frank, for giving the work around (here).

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s