How to Set Up a Linux Server from Scratch

Setting up a Linux server from scratch might seem daunting, but with the right guidance, anyone can master this essential IT skill. Whether you’re deploying a web server, database server, or development environment, understanding the fundamentals of Linux server setup will empower you to build robust, secure infrastructure. This comprehensive guide walks you through every step of the process, from choosing your distribution to securing your server.

Table of Contents

• Choosing the Right Linux Distribution
• Installing Your Linux Server
• Initial Server Configuration
• Setting Up User Accounts and Permissions
• Network Configuration
• Security Hardening Essentials
• Configuring the Firewall
• Updates and Maintenance
• Basic Monitoring Setup

Choosing the Right Linux Distribution

The first step to set up a Linux server is selecting the appropriate distribution. For server environments, stability and long-term support trump cutting-edge features. Ubuntu Server LTS (Long Term Support) and CentOS/Rocky Linux are popular choices for beginners due to their extensive documentation and community support.

Ubuntu Server offers five years of security updates and patches, making it ideal for production environments. Rocky Linux, the successor to CentOS, provides enterprise-grade stability. For learning purposes, platforms like DataCamp offer excellent courses on Linux fundamentals that can help you understand the differences between distributions.

Distribution Considerations

• Ubuntu Server: Beginner-friendly, extensive package repository, strong community
• Rocky Linux: Enterprise stability, RHEL compatibility, ideal for corporate environments
• Debian: Rock-solid stability, minimal resource usage, perfect for experienced administrators

Installing Your Linux Server

Once you’ve chosen your distribution, you’ll need hardware or a virtual private server (VPS). Cloud providers like Kamatera offer flexible VPS solutions perfect for deploying Linux servers with customizable resources.

Installation Steps

Download the ISO image from your chosen distribution’s official website. Create a bootable USB drive using tools like Rufus (Windows) or dd command (Linux). Boot from the installation media and follow these general steps:

1. Select your language and keyboard layout
2. Configure network settings (can be done post-installation)
3. Partition your disk (recommended: separate partitions for /, /home, and /var)
4. Set up your root password and create an administrative user
5. Select minimal installation for servers to reduce attack surface
6. Complete the installation and reboot

Initial Server Configuration

After installation, log in and perform these critical first steps:

Update System Packages

Always start with a fully updated system:

# For Ubuntu/Debian
sudo apt update && sudo apt upgrade -y

# For Rocky Linux/CentOS
sudo dnf update -y

Set the Hostname

Configure a meaningful hostname for your server:

sudo hostnamectl set-hostname your-server-name

Configure Timezone

Set the correct timezone for accurate logging:

sudo timedatectl set-timezone America/New_York
# List available timezones with: timedatectl list-timezones

Setting Up User Accounts and Permissions

Never use the root account for daily operations. Create a sudo-enabled user account:

# Create new user
sudo adduser username

# Add user to sudo group (Ubuntu)
sudo usermod -aG sudo username

# Add user to wheel group (Rocky Linux)
sudo usermod -aG wheel username

SSH Key Authentication

Implement SSH key-based authentication for enhanced security:

# On your local machine, generate SSH key pair
ssh-keygen -t ed25519 -C "your_email@example.com"

# Copy public key to server
ssh-copy-id username@server_ip

Network Configuration

Modern Linux distributions use different network management tools. Ubuntu uses Netplan, while Rocky Linux uses NetworkManager.

Ubuntu Netplan Configuration

Edit the Netplan configuration file at /etc/netplan/00-installer-config.yaml:

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: no
      addresses: [192.168.1.100/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]

Apply the configuration with:

sudo netplan apply

Security Hardening Essentials

Security should be your top priority when you set up a Linux server. Implement these hardening measures immediately:

Disable Root SSH Login

Edit /etc/ssh/sshd_config:

PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes

Restart SSH service:

sudo systemctl restart sshd

Install Fail2Ban

Protect against brute-force attacks:

sudo apt install fail2ban -y  # Ubuntu
sudo dnf install fail2ban -y  # Rocky Linux

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Configuring the Firewall

Ubuntu uses UFW (Uncomplicated Firewall), while Rocky Linux uses firewalld.

UFW Configuration (Ubuntu)

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Firewalld Configuration (Rocky Linux)

sudo systemctl start firewalld
sudo systemctl enable firewalld
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

Updates and Maintenance

Configure automatic security updates to keep your server protected:

Ubuntu Unattended Upgrades

sudo apt install unattended-upgrades -y
sudo dpkg-reconfigure --priority=low unattended-upgrades

Rocky Linux Automatic Updates

sudo dnf install dnf-automatic -y
sudo systemctl enable --now dnf-automatic.timer

Basic Monitoring Setup

Implement basic monitoring to track server health:

Install Essential Monitoring Tools

sudo apt install htop iotop nethogs -y  # Ubuntu
sudo dnf install htop iotop nethogs -y  # Rocky Linux

Check System Resources

Monitor your server regularly with these commands:

# CPU and memory usage
htop

# Disk usage
df -h

# Check running services
sudo systemctl list-units --type=service --state=running

# View recent logs
sudo journalctl -xe

Conclusion

Setting up a Linux server from scratch requires careful attention to detail, but following these steps ensures you build a secure, efficient foundation. Remember that server administration is an ongoing process—regular updates, monitoring, and security audits are essential for maintaining a healthy server environment.

Start with these fundamentals, and as you gain confidence, explore advanced topics like containerization, load balancing, and automated deployment pipelines. The skills you develop managing Linux servers will serve you throughout your IT career, whether you’re administering enterprise infrastructure or deploying personal projects.