# New Simplified Upload System

## Overview

This new upload system replaces the complex legacy upload flow with a cleaner, more reliable approach that provides clear status updates to users.

## File Structure

### New Files Created

1. **`upload_handler_new.php`** - Simplified upload handler
   - Validates files
   - Saves with proper naming: `shortname-shortdate.csv`
   - Logs to database
   - Triggers Python processing
   - Returns JSON response

2. **`get_processing_status.php`** - Status API endpoint
   - Returns detailed status information
   - Maps status codes to user-friendly messages
   - Includes progress percentage
   - Polled by frontend every 3 seconds

3. **`processing.php`** - Modern loading page
   - Beautiful UI with progress bar
   - Shows detailed status messages
   - Auto-redirects on completion
   - Handles errors and action required states
   - 7-minute timeout

## Upload Flow

```
1. User uploads file (cecl-portal.php)
   ↓
2. upload_handler_new.php receives file
   ↓
3. Validates file type, size
   ↓
4. Saves as: shortname-shortdate.csv in ../uploads/
   ↓
5. Updates database status to 'UPLOAD'
   ↓
6. Triggers Python script asynchronously
   ↓
7. Returns success JSON with redirect to processing.php
   ↓
8. processing.php polls get_processing_status.php every 3 seconds
   ↓
9. Shows real-time status updates with progress bar
   ↓
10. Auto-redirects when complete
```

## File Naming Convention

All uploaded files are now saved with a consistent naming pattern:

```
{shortname}-{shortdate}.csv
```

Examples:
- `samplebank-3q24.csv`
- `testbank-1q25.csv`
- `mybank-4q23.csv`

## Status Codes

The system uses the following status codes in the `ceclcompsettings.good` column:

### Processing Status
| Code | Message | Description | Progress |
|------|---------|-------------|----------|
| `V` | Processing report | Running CECL calculations | 50% |
| `Y` | Report ready! | CECL analysis completed successfully | 100% |
| `L` | Report ready! | Loan file successfully processed | 100% |

### Error Codes (requiring attention)
| Code | Message | Description |
|------|---------|-------------|
| `N` | Processing failed | General process failed |
| `B` | Blank codes detected | Contains blank call codes |
| `U` | Unlabeled codes detected | Contains unlabeled codes |
| `A` | Amortization error | Error in amortization calculation |
| `UP` | Database upload error | Error uploading table to database |
| `C` | Custom factors error | Error with custom factors |
| `R` | Data read error | Error reading data file |
| `D` | Duplicate column titles | File contains duplicate column titles |
| `CP` | Code pairs error | Error translating code pairs |
| `P` | Participation error | Error calculating participations |
| `FL` | Forward looking error | Error with forward looking adjustments |
| `VAR` | Variance threshold exceeded | Results outside variance threshold |
| `GOV` | Government guarantee error | Error processing government guarantees |

## Integration

### To use the new upload system:

1. **Update your upload form** to post to `upload_handler_new.php`:
   ```html
   <form action="upload_handler_new.php" method="POST" enctype="multipart/form-data">
       <input type="file" name="file" required>
       <input type="hidden" name="shortname" value="<?php echo $shortname; ?>">
       <input type="hidden" name="fileDate" value="<?php echo $fileDate; ?>">
       <input type="hidden" name="token" value="<?php echo $currentToken; ?>">
       <button type="submit">Upload</button>
   </form>
   ```

2. **Handle the JSON response** in your JavaScript:
   ```javascript
   fetch('upload_handler_new.php', {
       method: 'POST',
       body: formData
   })
   .then(response => response.json())
   .then(data => {
       if (data.status === 'success') {
           window.location.href = data.redirect_url;
       } else {
           alert(data.message);
       }
   });
   ```

## Benefits

✅ **Consistent file naming** - All files follow the same pattern  
✅ **Clear status updates** - Users see exactly what's happening  
✅ **Better error handling** - Detailed logging and user-friendly messages  
✅ **Modern UI** - Beautiful, responsive loading screen  
✅ **Auto-retry** - Continues polling until complete or timeout  
✅ **Simplified code** - Easier to maintain and debug  

## Debugging

### Log Files

- **`/cecl-app/upload_debug.log`** - Upload handler logs
- **PHP error_log** - Server-level errors

### Testing

To test the upload flow:

1. Upload a test file
2. Check the `upload_debug.log` for details
3. Monitor the `ceclcompsettings.good` column in the database
4. Watch the `processing.php` page for status updates

## Migration from Legacy System

The legacy system used:
- `ajaxTest.php` - Complex upload with inline processing
- `validateTitlesTest.php` - Mixed validation and status polling
- `loadingPage.php` - Basic spinner
- `checkReportStatus.php` - Simple status check

The new system simplifies this to:
- `upload_handler_new.php` - Clean upload handling
- `get_processing_status.php` - Detailed status API
- `processing.php` - Modern loading page

You can run both systems in parallel during migration by keeping the old files intact.

## Future Enhancements

- [ ] Email notifications on completion
- [ ] Resume processing from last checkpoint on failure
- [ ] Real-time Python script output streaming
- [ ] File upload progress bar
- [ ] Multiple file uploads (batch processing)