How to fix missing and underreplicated blocks - HDFS

-

$ su - <$hdfs_user> 

$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files

$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done

In above command, 3 is the replication factor. If you are using single datanode, it must be 1.


 




Popular posts from this blog

Shrink you container size up to 95%.

-
BLAFS is a bloat-aware filesystem for container debloating. The design principles of BLAFS are effective, efficient, and easy to use. It detects the files used by the container, and then debloats the container by removing the unused files. The debloated containers are still functional and can run the same workload as the original containers, but with a much smaller size and faster deployment. https://github.com/negativa-ai/BLAFS?tab=readme-ov-file
Read more

alma linux: dnf Module yaml error: Unexpected key in data

-
While applying recent updates to a server running Rocky Linux 8, an error presented itself. Module yaml error: Unexpected key in data: static_context [line 9 col 3] Module yaml error: Unexpected key in data: static_context [line 9 col 3] The solution turned out to be a simple one. Update the libmodulemd first to correct the problem then perform the dnf updates as usual. dnf update libmodulemd For a detailed explanation as to the cause of this problem.
Read more