Today my Nextcloud instance had a problem: It ran out of space.

I thought that wasn’t a problem because Nextcloud can be configured to use Amazon S3 for storage and I was already on an Amazon EC2 instance.

I didn’t want to build a whole new instance, my existing one had a lot of stuff in it.  I wanted to migrate the storage across to S3.  But apparently you can’t do that.

Challenge Accepted!

Nextcloud uses a completely different file naming convention when storing files locally to when it stores files on S3.  So to be able to migrate you will need some basic skills with:

  • Linux (or whatever operating system is hosting)
  • Mysql (or whatever database is your back end)
  • AWS S3 (being good with google is sufficient)

Before you start

Before you start you should:

  • Create a new S3 Bucket (note the bucket name and region)
  • Create an IAM user with read/write access to your new S3 bucket
  • Create security credentials to that user (note the key and secret)

THEN Create a completely new test site to work with.  Nextcloud falls over horribly if S3 is not configured well.  You don’t want to mix up a migration with bad configuration.  So sort your configuration out first.

I read about the configuration from here

Open the home page to your (un-configured instance).  Then edit the config file config/config.php and add the following:

'objectstore' => array(
        'class' => 'OC\\Files\\ObjectStore\\S3',
        'arguments' => array(
                'bucket' => 'nextcloud', // your bucket name
                'autocreate' => true,
                'key'    => 'EJ39ITYZEUH5BGWDRUFY', // your key
                'secret' => 'M5MrXTRjkyMaxXPe2FRXMTfTfbKEnZCu+7uRTVSj', // your secret
                'use_ssl' => true,
                'region' => 'eu-west-2', // your region
        ),
),

Steps to Migrate

1 – Kick off your users

Several steps here assume nothing has changed while you were working so DO NOT try to do this while users are on Nextcloud.

2 – Backup your instance

I take no responsibility for any damage done by following my instructions.  Backup your instance.  You have been warned!

3 – Rearrange your data directory

Don’t move anything, just create symbolic links. That way if this goes wrong you’ve done no damage.

My Data directory was /srv/nextcloud.  My Nextcloud database is called nextcloud_db  You will need to change these to match your own setup. When you are done you should end up with a directory full of files named something like urn:oid:1234.  At the command line I typed the following:

# Get the user files
mysql -p -B --disable-column-names -D nextcloud_db << EOF > user_file_list
   select concat('urn:oid:', fileid, ' ', '/srv/nextcloud/',
                 substring(id from 7), '/', path)
     from oc_filecache
     join oc_storages
      on storage = numeric_id
   where id like 'home::%'
   order by id;
EOF

# Get the meta files
mysql -p -B --disable-column-names -D nextcloud_db << EOF > meta_file_list
   select concat('urn:oid:', fileid, ' ', substring(id from 8), path)
     from oc_filecache
     join oc_storages
       on storage = numeric_id
     where id like 'local::%'
     order by id;
EOF

mkdir s3_files
cd s3_files
while read target source ; do
    if [ -f "$source" ] ; then
        ln -s "$source" "$target"
    fi
done < ../user_file_list

while read target source ; do
    if [ -f "$source" ] ; then
        ln -s "$source" "$target"
    fi
done < ../meta_file_list

4 – Copy your files to S3

If you have not already done so you will need to install the AWS CLI tools.  On Ubuntu this was simple enough:

apt-get install awscli
aws configure

Actually copying can take some time, particularly if you have > 10GB.

aws s3 sync . s3://my-nextcloud-bucket-name

5 – Update Nextcloud to Use S3.

STOP!

This step can damage your nextcloud instance!

Previous steps do not damage or change your Nextcloud instance because you were working on a copy of everything. Now you are about to change your existing Nexcloud instance.

Please don’t follow these steps blindly. Take the time to understand what you are doing first.

Again on the command line type:

mysql -p -D nextcloud_db << EOF > update oc_storages
      set id = concat('object::user:', substring(id from 7)) 
    where id like 'home::%';
   update oc_storages 
     set id = 'object::store:amazon::my-nextcloud-bucket'
   where id like 'local::%';
EOF

Then add the the extra configuration item to config/config.php

'objectstore' => array(
        'class' => 'OC\\Files\\ObjectStore\\S3',
        'arguments' => array(
                'bucket' => 'nextcloud', // your bucket name
                'autocreate' => true,
                'key'    => 'EJ39ITYZEUH5BGWDRUFY', // your key
                'secret' => 'M5MrXTRjkyMaxXPe2FRXMTfTfbKEnZCu+7uRTVSj', // your secret
                'use_ssl' => true,
                'region' => 'eu-west-2', // your region
        ),
),

Good to Go!

At this point you should be all set to use your instance again. Login yourself to check. Then let your users back in!