Prerequisites

This article builds on the prior article: Node Reference - History.

Deleting

Humans make mistakes. Your users are humans, and sometimes they will create a product by mistake and will need to delete it. For that scenario, we need a DELETE REST endpoint.

There are two main classes of delete actions: a “hard” delete and a “soft” delete. In a “hard” delete, the record is physically removed from the data store. In a “soft” delete, the record remains in tact but is marked as “deleted” so that it can be filtered out of subsequent searches. Each of these two classes has their own advantages and disadvantages depending on the situation.

Hard deletes have two main advantages:

• They are simple to implement. In our application we can simply issue a DyanamoDB delete call.
• The hard-deleted record does not take up any additional space in the data store’s table, and the record does not impact query performance. (As a DynamoDB table or index grows, the Scan operation slows down since every item is examined in a Scan.)

The big drawbacks of this approach are probably obvious:

There is no easy way to determine when the record was deleted, who deleted it, or a way to undelete it. One may think that our snapshotting strategy implemented in the prior history article would handle this. However, the snapshot stores the state of the record before the record was updated so in a delete scenario, the datetime and user who performed the delete would not be present in the snapshot table.

Soft deletes also have two main advantages:

• The flagging of a record as deleted is no different than any other update, so history tracking continues to work.
• Undeleting a record simply involves removing the “deleted” flag.

These benefits are, however, not free. There are several drawbacks to a soft-delete strategy:

• Deleted records still take up space in the data store. The file size isn’t much of a concern, but if a significant number of records are deleted, then query and scan performance could be impacted (when filtering out “deleted” records).
• Something has to filter the deleted records. The clients can check the deleted flag. However, this approach would require every client to consider and explicitly check for this flag. It is highly likely that other development teams may forget to do this non-intuitive step and cause incorrect results or confusion. This approach would also result in our service returning to the client potentially much larger results sets when including all historical data.
• We can filter out the records on the server side. This requires that every operation that loads a record for any reason check this “deleted” flag.

For our RESTful service, due to the fact that we have a need to support history, and because products will probably be rarely deleted, we are going to implement a soft delete strategy.

First, we will create our delete handler. Create a products/deleteProduct.spec.js test file with these contents:

const proxyquire = require('proxyquire');

describe('products', function () {
    
    describe('deleteProduct', function () {
        
        beforeEach(function () {
            process.env.PRODUCTS_TABLE_NAME = 'Products';
            this.response = {
            }
            this.getResponse = {
                Item: {
                    lastModified: '2018-01-02T03:04:05.000Z'
                }
            };

            this.context = {
                params: {}
            };

            const documentClient = this.documentClient ={
                get: () => ({
                    promise: () => Promise.resolve(this.getResponse)
                }),
                update: () => ({
                    promise: () => Promise.resolve({})
                })
            };
            spyOn(this.documentClient, 'get').and.callThrough();
            spyOn(this.documentClient, 'update').and.callThrough();

            this.snapshotProduct = (product) => Promise.resolve();
            spyOn(this, 'snapshotProduct');

            this.deleteProduct = proxyquire('./deleteProduct', {
                'aws-sdk': {
                    DynamoDB: {
                        DocumentClient: function() {
                            return documentClient;
                        }
                    }
                },
                './snapshots/snapshotProduct': this.snapshotProduct             
            });
        });

        afterEach(function() {
            jasmine.clock().uninstall();
        });

        it('should use the correct parameters to get the current state of the product', async function() {
            this.context.params.id = 'abc';

            await this.deleteProduct(this.context);
            
            const expectedParams = {
                TableName: 'Products',
                Key: {
                    id: 'abc'
                }
            };
            expect(this.documentClient.get.calls.argsFor(0)[0]).toEqual(expectedParams);
        });

        it('should pass the product to snapshotProduct', async function () {
            await this.deleteProduct(this.context);

            expect(this.snapshotProduct.calls.argsFor(0)[0]).toEqual(this.getResponse.Item);
        });

        it('should pass the correct TableName to documentClient.update', async function () {
            await this.deleteProduct(this.context);

            expect(this.documentClient.update.calls.argsFor(0)[0].TableName).toEqual('Products');
        });

        it('should pass the id to documentClient.update', async function() {
            this.context.params.id = 'abc';
            await this.deleteProduct(this.context);
            expect(this.documentClient.update.calls.argsFor(0)[0].Key.id).toEqual('abc');
        });

        it('should set the condition expression flag to true', async function() {
            await this.deleteProduct(this.context);

            expect(this.documentClient.update.calls.argsFor(0)[0].UpdateExpression)
                .toEqual('set deleted=:deleted, lastModified=:newLastModified');
        });

        it('should be a conditional update', async function () {
            await this.deleteProduct(this.context);
            expect(this.documentClient.update.calls.argsFor(0)[0].ConditionExpression).toEqual('lastModified = :lastModified');
        });
        
        it('should set the ExpressionAttributeValues', async function() {
            jasmine.clock().mockDate(new Date(Date.UTC(2018, 03, 05, 06, 07, 08, 100)));
            
            await this.deleteProduct(this.context);

            const expectedValues = {
                ':deleted': true,
                ':lastModified': '2018-01-02T03:04:05.000Z',
                ':newLastModified': '2018-04-05T06:07:08.100Z'
            };
            expect(this.documentClient.update.calls.argsFor(0)[0].ExpressionAttributeValues)
                .toEqual(expectedValues);
        });

        it('should set the status to 204 (no content)', async function (){
            await this.deleteProduct(this.context);
            expect(this.context.status).toEqual(204);
        });

        it('should return a 409 status if dynamo throws a constraint exception', async function () {
            const checkFailedError = {
                name: 'ConditionalCheckFailedException'
            };
            this.documentClient.update.and.returnValue({
                promise: () => Promise.reject(checkFailedError)
            });
            await this.deleteProduct(this.context);
            expect(this.context.status).toEqual(409);
        });
    });
});

We need to load the product, snapshot its current state, set the lastModified date, and save it using optimistic concurrency control just like our update endpoint. Add the following to products/deleteProduct.js:

const AWS = require('aws-sdk');
const documentClient = new AWS.DynamoDB.DocumentClient();
const snapshotProduct = require('./snapshots/snapshotProduct');
const productsTableName = process.env.PRODUCTS_TABLE_NAME || 'Products';

async function loadProduct(id) {
    const result = await documentClient.get({
        TableName: productsTableName,
        Key: {id}
    }).promise();
    return result.Item;
}

async function setDeleted(id, lastModified) {
    try {
        const newLastModified = (new Date(Date.now())).toISOString();
        await documentClient.update({
            TableName: productsTableName,
            Key: {
                id
            },
            ConditionExpression: 'lastModified = :lastModified',
            UpdateExpression: 'set deleted=:deleted, lastModified=:newLastModified',
            ExpressionAttributeValues: {
                ':lastModified': lastModified,
                ':deleted': true,
                ':newLastModified': newLastModified
            }
        }).promise();
    } catch (e) {
        if (e.name === 'ConditionalCheckFailedException') {
            return 409;
        }
        throw e;
    }
    return 204;
}

module.exports = async function(ctx) {
    const id = ctx.params.id;
    const product = await loadProduct(id);
    await snapshotProduct({...product});
    const lastModified = product.lastModified;

    ctx.status = await setDeleted(id, lastModified);
};

Add the delete route to server.js:

router.delete('/products/:id', require('./products/deleteProduct'));

We aren’t done yet. We now have to work our way through most of the other endpoints (e.g. GET, PATCH) and filter out the deleted record(s).

We will start by adding some test cases to products/listProducts.spec.js:

it('should filter out deleted items', async function() {
    await this.listProducts(this.context);
    expect(this.documentClient.scan.calls.argsFor(0)[0].FilterExpression)
        .toEqual('attribute_not_exists(deleted)');
});

We are leveraging a DynamoDB Filter Expression. A filter expression is not an index. It is equivalent to using an Array filter. It does save a bit of bandwidth by not having to ship filtered items across the network. Add the expected filter expression (FilterExpression: 'attribute_not_exists(deleted)') to products/listProducts.js so that your getProductList function looks like this:

module.exports = async function getProductList(ctx) {
    const scanOutput = await documentClient.scan({
        TableName: productsTableName,
        Limit: 25,
        ExclusiveStartKey: getExclusiveStartKey(ctx),
        FilterExpression: 'attribute_not_exists(deleted)'
    }).promise();

    addLinkHeaderIfNeeded(ctx, scanOutput.LastEvaluatedKey);
    ctx.body = scanOutput.Items;
};

Next, add a new test to products/getProductById.spec.js (that we created in Node Reference - Get Product by Id ) to return an HTTP 410 if the product has been deleted:

it('should return a 410 if the product has been deleted', async function () {
    this.response.Item.deleted = true;

    await this.getProductById(this.context);

    expect(this.context.status).toEqual(410);
});

We could have returned a 404 status code here and it would have been accurate. However, if we look at a list of standard HTTP status codes we can see that a 410 (Gone) method is more appropriate. There are a lot more status codes than most people realize and using the most specific status code can help clients better understand the effects of their request.

We also need similar test cases for the PATCH endpoint. Add these tests to products/updateProduct.spec.js:

describe('product has been deleted', function () {
    beforeEach(function() {
        this.getResponse.Item.deleted = true;
    });

    it('should return a 410 response', async function () {
        await this.updateProduct(this.context);

        expect(this.context.status).toEqual(410);
    });

    it('should not call documentClient.put', async function () {
        await this.updateProduct(this.context);

        expect(this.documentClient.put).not.toHaveBeenCalled();
    });
});

Implement the above test cases in products/getProductById.js and products/updateProduct.js and we will have a minimally viable soft delete strategy.

getProductById.js

Replace ctx.body = result.Item; with:

    if (!result.Item) {
        ctx.status = 404;
    } else if (result.Item.deleted) {
        ctx.status = 410;
    } else {
        ctx.body = result.Item;
    }

updateProduct.js

Add the following after const product = await loadProduct(id);:

    if (product.deleted) {
        ctx.status = 410;
        return;
    }

See the changes we made here.

Table of Contents

• Introduction
• Unit Testing
• Koa
• Docker
• Cloudformation
• CodePipeline
• Fargate
• Application Load Balancer
• HTTPS/DNS
• Cognito
• Authentication
• DynamoDB
• Put Product
• Validation
• Smoke Testing
• Monitoring
• List Products
• Get Product
• Patch Product
• History Tracking
• Delete (this post)
• Change Events
• Conclusion

If you have questions or feedback on this series, contact the authors at nodereference@sourceallies.com.