/project File System Issue

Posted on May 13, 2025 by Dani Cannella

1 Technical information on the recent /project file system issue.

2 Current status

The data recovery company has completed their analysis of the failed /project hard drives. We will be reaching out to PIs individually to discuss the next steps.

3 Updates

Update (6/9/25): We have identified complete lists of affected files with false positives filtered out. There is a text file in each directory containing a list of affected files with the naming format of CORRUPT_20250606__project_ttrojan_120.txt. Only users assigned to that project can view the list. The previous lists (that contained false positives) have not been removed from the project directories, they’ve just been hidden.

Update (5/15/25): Please be aware that snapshots were also affected by the /project disk failure. At this time, snapshots are not a viable method of data recovery for users whose data was affected.

Additionally, copying data that was stored on the affected disks to either /scratch1 or /project2 will not solve the issue. We are seeing runtime issues on /scratch1 from attempts to copy over lost or corrupted data from the failed /project disks. We know this is very stressful for many of you, but we ask that you please be patient while the data recovery company works to resolve this issue.

4 Timeline

April 27, 2025 @ 10:31 AM PDT: A routine drive failure occurred in one of the storage pools.

The way /project is configured is meant to safeguard against disk failures. Each file on /project is broken up into chunks and stored on 4 of 80 total storage pools. Each storage pool is made up of an array of 12 hard drives that can tolerate the loss of 2 hard drives while still maintaining the data in the pool. Hard drive failures are not uncommon and we routinely replace failed drives. Once the data from a failed drive is copied over to the new drive, the pool is fully functioning again and back to full fault tolerance.

April 28, 2025 @ 8:05 AM PDT: We started a routine rebuild operation for the failed drive.

April 30, 2025 @ 9:39 AM PDT: During the single failed drive’s replacement, two more drives in the same pool failed, resulting in 3 total failed drives simultaneously occurring during the rebuild. This is a very rare occurrence.

At this point, many users started receiving remote I/O errors.

May 7, 2025: After a series of troubleshooting measures and an emergency maintenance, the failed drives were removed from the file system and sent to a data recovery company.

At this point, many users started receiving incomplete file errors.

5 Check your data status

The previous methods for identifying affected files contained false positives. Not every file that was assigned to storage target 55 was affected.

The most accurate way to determine if your files have been affected is to check the lists of files that have been generated for each /project allocation. To find the list, navigate to your project directories. The path will look something like /project/ttrojan_120/INCOMPLETE_20250606__project_ttrojan_120.txt. Lists are specific to the project directory they are in and can only be accessed by users assigned to that project.

If you discover that your data has been affected by the /project disk failure, please email us at carc-support@usc.edu or submit a help ticket.

6 Cold Storage System and data migration

While we do offer snapshots for /project and /project2, they are not the most reliable way to back up your data. CARC offers the Cold Storage System as a solution for long-term data archiving for large data sets. PIs can request a cold storage allocation via the CARC user portal.

For more information on pricing and how to use this system, see our CARC Cold Storage System guide.

Additionally, we are planning to migrate all projects to the new /project2 file system. The /project2 storage system is built entirely from solid state drives (SSDs), offering significantly better performance, speed, and dependability. We are aiming to start this process in late July or early August, 2025. We will keep you updated on this as it gets closer.

For more information and pricing, please see our Project file system guide.

7 Frequently Asked Questions

How much data has been lost?

Approximately 5% of data has been lost uniformly across the /project file system. This translates to approximately 400 TB of data.

I saw your email statement. What does “some data was permanently lost” mean?

We apologize for the unclear language. “Some data” referred to the /project file system as a whole. All incomplete files have been lost.

What is the /project/ttrojan_123/CORRUPT_20250606__project_ttrojan_123.txt file?

This text file contains a list of the files and symlinks that were corrupted in the incident.

If the file is empty, no corrupt files were detected in the /project/ttrojan_123/ hierarchy.

Please note that the /project/ttrojan_123/INCOMPLETE_20250606__project_ttrojan_123.txt is no longer up-to-date.

What is the format of the /project/ttrojan_123/CORRUPT_20250606__project_ttrojan_123.txt files?

The format is 5 fields separated by a space:

  • Field 1: file size in bytes as reported by the ls command
  • Field 2: username or uid of the owner of the file or symlink
  • Field 3: groupname or gid of the group owner of the file or symlink
  • Field 4: 1 letter code for the type of file (f = regular file, l = symlink)
  • Field 5: the fully qualified file path of the corrupted file

What is the deadline to transfer data from /project to /project2?

The /project file system will be decommissioned by January 1, 2026. All data on /project should be transferred to /project2 by November 1, 2025. After that date, data on /project will be read-only. All data on /project will be permanently lost after December 31, 2025.

The CARC team is here to assist you with migrating your data, including providing a script you can use to automate the process. If you have any questions about the migration or need assistance, please don’t hesitate to contact us at carc-support@usc.edu or submit a help ticket.

I’m a Laguna user. Has my data been lost?

No, Laguna has its own dedicated storage file system. No Laguna data has been lost. However, we still strongly recommend keeping backups of your important files following the 3-2-1 strategy.