MDEV-37605 Extend mariadb-binlog to convert innodb based binary logs to legacy#5276
MDEV-37605 Extend mariadb-binlog to convert innodb based binary logs to legacy#5276tarunw07 wants to merge 4 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the --convert-engine-binlog option to mariadb-binlog for converting InnoDB-based engine binlog files into legacy-format binary logs, along with helper functions for event serialization and new test suites. The code review identified several critical issues, including potential NULL pointer dereferences of gtid_state, fdev, gtid_set, and list, as well as a memory leak on write failures in write_event_to_legacy_binlog. Additionally, the feedback highlights return type mismatches, inconsistent boolean usage, and multiple coding standard violations regarding spacing after if keywords.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if (ev->get_type_code() == GTID_EVENT) { | ||
| rpl_gtid ev_gtid; | ||
| Gtid_log_event *gle= (Gtid_log_event*) ev; | ||
| ev_gtid= {gle->domain_id, gle->server_id, gle->seq_no}; | ||
|
|
||
| gtid_state->update_nolock(&ev_gtid); | ||
| } |
There was a problem hiding this comment.
If gtid_state is not initialized (e.g., if the conversion starts or encounters a GTID_EVENT before gtid_state is allocated), dereferencing it via gtid_state->update_nolock(&ev_gtid) will cause a crash. We should ensure gtid_state is initialized or add a defensive NULL check.
if (ev->get_type_code() == GTID_EVENT) {
rpl_gtid ev_gtid;
Gtid_log_event *gle= (Gtid_log_event*) ev;
ev_gtid= {gle->domain_id, gle->server_id, gle->seq_no};
if (gtid_state)
gtid_state->update_nolock(&ev_gtid);
}| bool | ||
| Gtid_list_log_event::to_packet(String *packet) | ||
| { | ||
| uint32 i; | ||
| uchar *p; | ||
| uint32 needed_length; | ||
|
|
||
| DBUG_ASSERT(count < 1<<28); | ||
|
|
||
| needed_length= packet->length() + get_data_size(); | ||
| if (packet->reserve(needed_length)) | ||
| return true; | ||
| p= (uchar *)packet->ptr() + packet->length();; | ||
| packet->length(needed_length); | ||
| int4store(p, (count & ((1<<28)-1)) | gl_flags); | ||
| p += 4; | ||
| /* Initialise the padding for empty Gtid_list. */ | ||
| if (count == 0) | ||
| int2store(p, 0); | ||
| for (i= 0; i < count; ++i) | ||
| { | ||
| int4store(p, list[i].domain_id); | ||
| int4store(p+4, list[i].server_id); | ||
| int8store(p+8, list[i].seq_no); | ||
| p += 16; | ||
| } | ||
|
|
||
| return false; | ||
| } |
There was a problem hiding this comment.
If list is NULL (which can happen if my_malloc failed during construction or if the event is invalid), dereferencing list[i] in the loop will cause a crash. We should check is_valid() or list != NULL before proceeding. This also cleans up a double semicolon typo on line 2850.
bool
Gtid_list_log_event::to_packet(String *packet)
{
uint32 i;
uchar *p;
uint32 needed_length;
if (!is_valid())
return true;
DBUG_ASSERT(count < 1<<28);
needed_length= packet->length() + get_data_size();
if (packet->reserve(needed_length))
return true;
p= (uchar *)packet->ptr() + packet->length();
packet->length(needed_length);
int4store(p, (count & ((1<<28)-1)) | gl_flags);
p += 4;
/* Initialise the padding for empty Gtid_list. */
if (count == 0)
int2store(p, 0);
for (i= 0; i < count; ++i)
{
int4store(p, list[i].domain_id);
int4store(p+4, list[i].server_id);
int8store(p+8, list[i].seq_no);
p += 16;
}
return false;
}| Gtid_list_log_event::Gtid_list_log_event(rpl_binlog_state_base *gtid_set) | ||
| : count(gtid_set->count_nolock()), gl_flags(0), list(0), sub_id_list(0) | ||
| { | ||
| cache_type= EVENT_NO_CACHE; | ||
| /* Failure to allocate memory will be caught by is_valid() returning false. */ | ||
| if (count < (1<<28) && | ||
| (list = (rpl_gtid *)my_malloc(PSI_INSTRUMENT_ME, | ||
| count * sizeof(*list) + (count == 0), MYF(MY_WME)))) | ||
| { | ||
| gtid_set->get_gtid_list_nolock(list, count); | ||
| } | ||
| } |
There was a problem hiding this comment.
If gtid_set is NULL, calling gtid_set->count_nolock() in the initializer list will result in a NULL pointer dereference and crash. We should add a defensive check to handle a NULL gtid_set gracefully.
Gtid_list_log_event::Gtid_list_log_event(rpl_binlog_state_base *gtid_set)
: count(gtid_set ? gtid_set->count_nolock() : 0), gl_flags(0), list(0), sub_id_list(0)
{
cache_type= EVENT_NO_CACHE;
/* Failure to allocate memory will be caught by is_valid() returning false. */
if (gtid_set && count < (1<<28) &&
(list = (rpl_gtid *)my_malloc(PSI_INSTRUMENT_ME,
count * sizeof(*list) + (count == 0), MYF(MY_WME))))
{
gtid_set->get_gtid_list_nolock(list, count);
}
}
ParadoxV5
added
GSoC
External Contribution
e38182e to
cfd3e68
Compare
528c9e1 to
da82682
Compare
…e ROTATE_EVENT whenever a binlog file is rotated.
| static bool init_output_legacy_binlog(FILE **out_file, char *out_name, | ||
| size_t out_name_len) | ||
| size_t out_name_len, bool rotate) | ||
| { | ||
| if (rotate) | ||
| { |
There was a problem hiding this comment.
Having a separate rotate_output_legacy_binlog() that calls init_output_legacy_binlog() sounds better than shoving rotation inside “reïnitialization”, right?
There was a problem hiding this comment.
Yes, thanks for your input. Your suggestion seems like a much better design, so I’ll go ahead and make these changes. Thanks! :)
88cc385 to
2251204
Compare
…n GTID_EVENT whenever the max binlog size is reached
2251204 to
937845a
Compare
| @@ -1 +1,2 @@ | |||
| --source include/not_embedded.inc | |||
There was a problem hiding this comment.
@ParadoxV5 @bnestere Can you review/confirm this change? mysqlbinlog_convert_engine_binlog_basic test was failing because it was running on embedded server. I could've added this line in the test but this felt like a better option.
There was a problem hiding this comment.
Hm, it makes sense.
Compare with have_log_bin.inc, which literally has:
source include/not_embedded.inc;
In your tests, all but mysqlbinlog_convert_engine_binlog_basic calls have_binlog_format_row.inc, which in turn calls have_log_bin.inc.
I see that most have_innodb_binlog.inc callers aren’t “affected” either because they transitively call have_log_bin.inc.
The other tests I found call not_embedded.inc directly, like your alternative solution; they are:
binlog_in_engine.mariabackup_binlog_dirbinlog_in_engine.mariabackup_binlogsmariabackup_slave_provision.inccallers
I wonder if it even makes sense to include the entire have_log_bin.inc in have_innodb_binlog.inc, not just --log-bin in have_innodb_binlog.opt?
| --source include/not_embedded.inc | |
| --source include/have_log_bin.inc |
Though a shortcoming with have_log_bin.inc is that it will run the test in each of the three content formats (Mixed (default), Statement, Row).
This is unnecessary for tests that are more about the container than the contained contents, including yours, hence the calls to have_binlog_format_X.inc.
There was a problem hiding this comment.
Could you elaborate a bit more on the shortcoming? I’m not sure I fully understood it.
My intention was for mysqlbinlog_convert_engine_binlog_basic to run with all three binlog formats: ROW, STATEMENT, and MIXED.
I did consider the change you suggested, but I was hesitant because have_log_bin.inc internally sources include/no_view_protocol.inc, and I wasn’t sure whether that could introduce any unexpected side effects. That said, logically, including have_log_bin.inc from have_innodb_binlog.inc does seem coherent.
There was a problem hiding this comment.
(answered in our fortnightly call)
3 participants
Summary
This is an initial work-in-progress patch for MDEV-37605: Convert InnoDB Binary Logs to Legacy Format.
The goal of this work is to extend
mariadb-binlogso that it can read InnoDB-based binary log files (.ibb) and generate legacy-format binary log files as output. This is intended to support compatibility use cases such as downgrades, migrations, and tooling that expects the legacy binlog format.The current approach is to avoid fully deserializing and reserializing regular events. Since most event contents are already encoded in the
.ibbfile, the converter should mostly copy event bytes into the generated legacy binlog, while handlingFORMAT_DESCRIPTION_EVENT,GTID_LIST_EVENT,BINLOG_CHECKPOINT_EVENT&ROTATE_EVENT.Current status
This PR currently includes:
--convert-engine-binlogFORMAT_DESCRIPTION_EVENThandlingGTID_LIST_EVENTBINLOG_CHECKPOINT_EVENTend_log_poscorrectly in generated legacy binlog events.ibbfile, converting it to legacy binlog format, and checking the converted outputWork still in progress
This PR is not ready for final review yet. I am opening it early so that discussion, progress tracking, CI feedback, and initial code review can happen on the PR.
The main remaining pieces are:
ROTATE_EVENTFSP_BINLOG_TYPE_GTID_STATEfrom the first page of the.ibbfile to initialize the converter’s GTID state.FORMAT_DESCRIPTION_EVENThandlingDesign notes
The current design direction is:
FORMAT_DESCRIPTION_EVENT,GTID_LIST_EVENT,BINLOG_CHECKPOINT_EVENT&ROTATE_EVENT) that must be synthesized for legacy binlog output.FSP_BINLOG_TYPE_GTID_STATEfrom the first page of the.ibbfile to initialize the converter’s GTID state.rpl_binlog_state_base, so thatGTID_LIST_EVENTcan be generated correctly at the beginning of each output legacy binlog file.BINLOG_CHECKPOINT_EVENTnear the beginning of each generated legacy binlog file.end_log_posfor all the generated legacy binlog events.