Posted on June 14, 2013 - by Sarath
I have been recently working on a command line client for Mega cloud storage service. If you are not familiar with mega, let me give you a brief intro. Mega (https://mega.co.nz) is a storage service founded by Kim Dotcom exactly one year after MegaUpload was raided by US government and seized his entire property. The case against him was that he hosted an upload service which was being misused by people and become the source of piracy. He recently came up with a nice storage solution out of New Zealand which encrypts data at client side and transfers to the server. At mega server side, nobody will be able to decrypt your data since it is encrypted at client side. Your password is the encryption key used for the data. The downside is that, once you forget the password you will not be able to reset your password or recover you data back.
I will run through a very quick overview of how a client interacts with Mega service and its architecture.
Mega uses json a the unit of API request and response. It uses the following request and response formats:
Response: [res1,res2,...] or error code -errcode or [-errcode]
A user has to initialize a user session by login with username and password. A login request consists of username and a custom hash(password). It responds with sessionId, RSA private key and user master key. All of them encrypted. User master key is a symmetric encryption key for decrypting keys specific to all the files stored by a user account. In order to decrypt the sessionId, first compute the password hash, and decrypt the master key using the password hash. Once master key is decrypted, use that to decrypt the RSA composite key. Composite RSA key contains p,q and d components (Read http://en.wikipedia.org/wiki/RSA_(algorithm)#Operation ), all of them encoded in multi-precision integer format. Once you decode these components, we can decrypt the sessionId. SessionId is also represented as multi precision int (m). Hence, sessionId = (m ^ d) % n where n = p*q. Since mega uses json for request and response, it should be ascii encoded. Hence, all binary data is represented as a specialized base64url encoding with padding characters (=) removed. Hence sessionId is converted into base64url encoding. Once we decode sessionId, it used used for all the mega file operations. For identifying the requests, mega requires a id to be passed along with requests which needs to be incremented everytime. Client requests are post to the url https://eu.api.mega.co.nz/cs?id=seqid&sid=sessId. Mega uses 128 bit block AES encryption all over the place.
Mega stores files in node hierarchical parent-child model like standard filesystems. Mega has three root nodes, either Mega Cloud, Trash or Inbox. Each node can be either a folder type, file type or root. Mega stores each node as a block of bytes with a size, timestamp, key, attribute, parent, node type identified by a hash. Attribute contains encrypted filename by using encrypted file key. User master key is used to decrypt the file key. Using the file key, file content and attribute can be decrypted. The file key is a composite key which contains additional initialization vector and meta-CBC-MAC (Message Authentication Code) data in case of a file node. The actual file content is encrypted in AES counter mode using this init vector. CBC-MAC meta data is used for integrity verification of the file.
Once you get a session ID, we can use file list APi to download meta data about all the nodes in the filesystem tree. The response contain, one metadata entry per node.
To download a file, we should send a request with node Hash. The response will contain a url at which we can download the file. The file should be downloaded in chunks of size, 128K / 384K / 768K / 1280K / 1920K / 2688K / 3584K / 4608K / … (every 1024 KB). This is to allow integrity checks per chunk. While downloading each chunk, it should be decrypted using AES-Counter mode using the file key.
To upload a file, we should generate a random key to be used as file key and initialization vector. An upload request with file size is issued and it returns a response with a url at which we can do uploads. Uploads are done in chunks as download by posting encrypted data at http://uploadurl/chunkstart-chunkend. Once all the uploads are complete, we receive a completion handle. We should post a file upload completion request with completion handle, meta-MAC, attributes, parent hash, etc to finalize the upload.
There are other operations like delete, move, update attribute, etc available.
Since we keep a local copy of entire filesystem, we need a way to update our local cache of filesystem when some other client using the same account updates the filesystem by adding, removing or updating files or folders. For eg, if we login to mega.co.nz account in two browsers with same credential, we can see that if we make any change on one browser that gets reflected in other as well. In order to provide filesystem change notification, mega provides a mechanism for server to talk to the client.
About megacmd utility..
Mega service provides a mechanism to access the Mega FS tree in a model with hash identifiers. But they are not readable and we need a path name representation using resource URIs. So I initially wrote a library for go called go-mega which provides Mega API abstraction in terms of hash and filesystem tree. Then, I wrote another go package called megaclient which abstracts in terms of file path and it provides similar operations as s3cmd. Code for both can be found in my github.
megacmd is a command-line utility written on top of megaclient package. The usage for megacmd is as follows:
megacmd [OPTIONS] list mega:/foo/bar/
megacmd [OPTIONS] get mega:/foo/file.txt /tmp/
megacmd [OPTIONS] put /tmp/hello.txt mega:/bar/
megacmd [OPTIONS] delete mega:/foo/bar
megacmd [OPTIONS] mkdir mega:/foo/bar
megacmd [OPTIONS] move mega:/foo/file.txt mega:/bar/foo.txt
megacmd [OPTIONS] sync mega:/foo/ /tmp/foo/
megacmd [OPTIONS] sync /tmp/foo mega:/foo
-conf="/Users/slakshman/.megacmd.json": Config file path
-force=false: Force hard delete or overwrite
-ignore-same-size=false: Consider files with same size and path suffix as same
-recursive=false: Recursive listing
Downloadable binaries for GNU/Linux and Mac OSX are available.
Please read more and find documentation at https://github.com/t3rm1n4l/megacmd
If you find any bugs or would like to contribute to the project, please feel free to file an issue or send a pull request at github.