Jason Hoffman, CTO and founder of cloud service provider Joyent, details how combining object storage and parallel compute clusters can make working with big data easier and faster by eliminating bottlenecks.
How objects and compute will eat the world
Networked storage vendors’ days are numbered. Customers are fleeing to consolidated online object storage, and soon the digital object storage will surpass traditional file storage as the primary model for data outside of a DBMS. But there’s a subtle and often unappreciated downside to most distributed object storage: data inertia. The implicit limits on moving huge data sets to in-network compute nodes deter business or clinical insights from surfacing.
At Joyent, we architected the Manta Storage and Compute Service — “Manta,” for short — to be a best-in-class object store and an in-storage massively parallel compute cluster. It drives data latency effectively to zero, moving weekly or monthly jobs to an hourly or even an on-demand analytic cadence.