Can not train

[Tshiyao]

How can i deal with this?What should I do?

what is SynSQL-2.5M/databases/version_control_and_source_code_management?

‘’‘bash
File "/data/miniconda/envs/sqlr1/lib/python3.9/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
File "/data/miniconda/envs/sqlr1/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
File "/data/miniconda/envs/sqlr1/lib/python3.9/site-packages/ray/_private/worker.py", line 2858, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/data/miniconda/envs/sqlr1/lib/python3.9/site-packages/ray/_private/worker.py", line 958, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(FileNotFoundError): ray::main_task() (pid=34103, ip=10.230.62.222)
File "/data/coding/SQL-R1/verl/trainer/main_ppo.py", line 185, in main_task
trainer.fit()
File "/data/coding/SQL-R1/verl/trainer/ppo/ray_trainer.py", line 600, in fit
val_metrics = self._validate()
File "/data/coding/SQL-R1/verl/trainer/ppo/ray_trainer.py", line 453, in _validate
reward_tensor = self.val_reward_fn(test_batch)
File "/data/coding/SQL-R1/verl/trainer/main_ppo.py", line 76, in call
score = compute_score_fn(solution_str=sequences_str, ground_truth=ground_truth)
File "/data/coding/SQL-R1/verl/utils/reward_score/synsql.py", line 166, in compute_score
exec_status = func_timeout(
File "/data/miniconda/envs/sqlr1/lib/python3.9/site-packages/func_timeout/dafunc.py", line 108, in func_timeout
raise_exception(exception)
File "/data/miniconda/envs/sqlr1/lib/python3.9/site-packages/func_timeout/py3_raise.py", line 7, in raise_exception
raise exception[0] from None
File "/data/coding/SQL-R1/verl/utils/reward_score/exec_eval.py", line 200, in eval_exec_match
db_paths = [os.path.join(db_dir, basename) for basename in os.listdir(db_dir) if '.sqlite' in basename]
FileNotFoundError: [Errno 2] No such file or directory: 'data/NL2SQL/SynSQL-2.5M/databases/version_control_and_source_code_management'

'''

[MPX0222]

This is the name of SynSQL database. Maybe you need to download and unzip the related database file.

[Tshiyao]

Thank you for your reply!
But I followed the instructions in the readme, why are files missing?
Could you please provide more detailed directory contents?

And why does the reward function reward_score/exec_eval.py in Verl require SynSQL data?

[MPX0222]

exec_eval.py needs to read the database file to get the execution feedback. So SynSQL database files are necessary for training process.

[Tshiyao]

I used the following script to run the training:

export WANDB_API_KEY=your_wandb_api_key
# export VLLM_ATTENTION_BACKEND=XFORMERS
export VLLM_ATTENTION_BACKEND=FLASH_ATTN

DATA_DIR_PATH=data

RUN_ID=0.5B
GPU_ENV=1GPU
MODEL_ENV=Qwen2.5-Coder-0.5B-Instruct
PROJECT_NAME=SQL-R1
        
LOG_PATH=logs/$PROJECT_NAME
MODEL_PATH=models/$MODEL_ENV
EXPERIMENT_NAME=$GPU_ENV-$MODEL_ENV-$RUN_ID

mkdir -p $LOG_PATH

set -x

nvidia-smi

python -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=$DATA_DIR_PATH/train.parquet \
    data.val_files=$DATA_DIR_PATH/test.parquet \
    data.train_batch_size=8 \
    data.val_batch_size=8 \
    data.max_prompt_length=4096 \
    data.max_response_length=2048 \
    actor_rollout_ref.model.path=$MODEL_PATH \
    actor_rollout_ref.actor.optim.lr=3e-7 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=8 \
    actor_rollout_ref.actor.ppo_micro_batch_size=8 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=True \
    actor_rollout_ref.actor.fsdp_config.grad_offload=True \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
    actor_rollout_ref.rollout.log_prob_micro_batch_size=80 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.2 \
    actor_rollout_ref.rollout.n=8 \
    actor_rollout_ref.rollout.temperature=1.1 \
    actor_rollout_ref.ref.log_prob_micro_batch_size=80 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.kl_ctrl.kl_coef=0.001 \
    trainer.critic_warmup=0 \
    trainer.logger=['console'] \
    trainer.project_name=$PROJECT_NAME \
    trainer.experiment_name=$EXPERIMENT_NAME \
    trainer.n_gpus_per_node=1 \
    trainer.nnodes=1 \
    trainer.default_local_dir=$LOG_PATH/$EXPERIMENT_NAME \
    trainer.default_hdfs_dir=null \
    trainer.save_freq=100 \
    trainer.test_freq=100 \
    trainer.total_epochs=10 $@ 2>&1 | tee $LOG_PATH/$MODEL_ENV/grpo.log

However, instead of starting the training, the script exits after generating the following output:

oftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>
(main_task pid=3301) 
(main_task pid=3301) [Structure Validation]
(main_task pid=3301)   <think>: count=1, position=1
(main_task pid=3301)   </think>: count=1, position=1399
(main_task pid=3301)   <answer>: count=1, position=1408
(main_task pid=3301)   </answer>: count=1, position=2060
(main_task pid=3301) Tag sequence validation passed
(main_task pid=3301) [Parsed SQL]: SELECT u.role FROM users u JOIN access_logs al ON u.user_id = al.user_id WHERE al.error_count = 0 AND al.resolved_count = 0 GROUP BY u.role ORDER BY AVG(al.completeness) DESC;
(main_task pid=3301) 
(main_task pid=3301) [Format validation] PASS
(main_task pid=3301) [Format score]: 1
(main_task pid=3301) [DB NAME]: biomedical_data_management_and_research_collaboration
(main_task pid=3301) [Gold SQL]: WITH UserAccessCounts AS ( -- Count the number of unique datasets accessed by each user SELECT user_id, COUNT(DISTINCT dataset_id) AS dataset_count FROM access_logs GROUP BY user_id HAVING COUNT(DISTINCT dataset_id) > 1 -- Users with more than one dataset access ), ErrorFreeDatasets AS ( -- Get datasets that have no errors or resolved errors SELECT d.dataset_id, d.completeness FROM datasets d LEFT JOIN errors e ON d.dataset_id = e.dataset_id WHERE e.error_id IS NULL OR (e.resolved = 1 AND e.resolved_date IS NOT NULL) ), AccessedDatasets AS ( -- Join access logs with user access counts and error-free datasets SELECT al.user_id, u.role, ef.dataset_id, ef.completeness FROM access_logs al JOIN UserAccessCounts uac ON al.user_id = uac.user_id JOIN users u ON al.user_id = u.user_id JOIN ErrorFreeDatasets ef ON al.dataset_id = ef.dataset_id ) -- Calculate average completeness and order by average completeness descending SELECT role, AVG(completeness) AS avg_completeness FROM AccessedDatasets GROUP BY role ORDER BY avg_completeness DESC;
(main_task pid=3301) [Pred SQL]: SELECT u.role FROM users u JOIN access_logs al ON u.user_id = al.user_id WHERE al.error_count = 0 AND al.resolved_count = 0 GROUP BY u.role ORDER BY AVG(al.completeness) DESC;
(main_task pid=3301) [g_flag]: exception
(main_task pid=3301) [p_flag]: exception
(main_task pid=3301) [g_denotation]: incomplete input
(main_task pid=3301) [p_denotation]: no such column: al.error_count
(main_task pid=3301) [Exec status]: Unexecutable
(main_task pid=3301) 
(main_task pid=3301) --------------------------------------------------------------------------------
(main_task pid=3301) --------------------------------- Final Score ----------------------------------
(main_task pid=3301)  -- Format Score: 1
(main_task pid=3301)  -- Exec Score: -2

Could you help me figure out why the training isn't starting and the script exits after generating this output?

[MPX0222]

It seems that you just use 1 GPU to run training script. Could you tell me the model and quantity of the GPU you are using?

If you need to train with this script, it is usually recommended to have 4 GPU with 48GB memory (Just for <7B model) or more for training. Training with only 1 GPU can easily lead to memory overflow during the rollout process, that is, an error interrupt occurs in the process you are demonstrating.

[Tshiyao]

I used 1xA100-40GB and Qwen2.5-0.5b-Coder to verify if the script could run. Strangely, after the rollout process, the script exited without any error messages.

[MPX0222]

This is normal. Usually, it is due to insufficient memory causing interruption. It is recommended that you use more GPU for training.