MongoDB Transaction BulkWrite Endless Retry

Previously we talked about # How to Retry MongoDB Transaction. However, if you use BulkWrite() and one of the operation is retryable (e.g. duplicated key error), the new transactions API will retry the bulk write endlessly which might lead to server CPU 100%. (MongoDB Server v4.4.6-ent, MongoDB Driver v2.12.2)

To avoid such issue, we have three suggestions:

• Add a cancellation token to limit the max retry time
• Break the transaction after max retry count
• Set BulkWriteOptions { IsOrdered = true }

The first two suggestions are also applicable to transactions which don't use BulkWrite().

Limit Max Transaction Execution Time

First, let's refine the RetryReplaceAsync() by adding CancellationToken, which limits the max execution time of the transaction:

private async Task RetryReplaceAsync(IMongoClient mongoClient, string uuid, string value)
{
	var collection = mongoClient.GetDatabase(DatabaseName).GetCollection<BsonDocument>(CollectionName);

	using (var session = await mongoClient.StartSessionAsync())
	using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(1)))
	{
		var filter = Builders<BsonDocument>.Filter.Eq("Uuid", uuid);

		await session.WithTransactionAsync(
			async (s, ct) =>
			{
				await collection.ReplaceOneAsync(s, filter, new BsonDocument { { "Uuid", uuid }, { "op", value } }, new ReplaceOptions { IsUpsert = true }, ct);
				return string.Empty;
			},  cancellationToken: cts.Token);
	}
}

Limit Max Transaction Retry Count

Just add 3 lines:

private async Task RetryReplaceAsync(IMongoClient mongoClient, string uuid, string value)
{
    var collection = mongoClient.GetDatabase(DatabaseName).GetCollection<BsonDocument>(CollectionName);

    int count = 0;
    using (var session = await mongoClient.StartSessionAsync())
    using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(1)))
    {
        var filter = Builders<BsonDocument>.Filter.Eq("Uuid", uuid);

        await session.WithTransactionAsync(
            async (s, ct) =>
            {
                if (++count >= 3)
                {
                    throw new ApplicationException($"Reached max retry times");
                }
                await collection.ReplaceOneAsync(s, filter, new BsonDocument { { "Uuid", uuid }, { "op", value } }, new ReplaceOptions { IsUpsert = true }, ct);
                return string.Empty;
            },  cancellationToken: cts.Token);
    }
}

Execute BulkWrite Operations In Order

The bulk write operation can be executed ordered or unordered. To improve performance, it seems that unordered execution is better. However, the unordered execution might eat some exceptions for a single operation, which leads to endless transaction retry.

As a result, we suggest to execute the operations in order. The code looks like this:

await collection.BulkWriteAsync(session, listWrites, new BulkWriteOptions { IsOrdered = true }, cancellationToken: ct);

Reference

https://developer.mongodb.com/community/forums/t/handling-duplicated-key-error-in-bulk-insert-retry-scenarios/2869

https://stackoverflow.com/questions/61244296/how-to-handle-duplicate-error-in-mongo-bulkinsert-retry