1 From 2a6f0dd5425cf43b8c09a8203e6ee64ba2b3868d Mon Sep 17 00:00:00 2001
2 From: Bogdan Hamciuc <bogdan.hamciuc@nxp.com>
3 Date: Tue, 12 Jan 2016 08:58:40 +0200
4 Subject: [PATCH 202/226] staging: fsl-dpaa2: eth: code cleanup for
7 -this is a squash of cleanup commits (see QLINUX-5338), all commit logs
10 Signed-off-by: Stuart Yoder <stuart.yoder@nxp.com>
12 ---------------------------------------------------------------------
14 fsl-dpaa2: eth: Drain queues upon ifconfig down
16 MC firmware assists in draining the Tx FQs; the Eth driver flushes the
17 Rx and TxConfirm queues then empties the buffer pool.
19 Signed-off-by: Bogdan Hamciuc <bogdan.hamciuc@nxp.com>
21 fsl-dpaa2: eth: Don't use magic numbers
23 Add a define to avoid mentioning directly the maximum number
24 of buffers released/acquired through a single QBMan command.
26 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
28 dpaa2-eth: Remove cpumask_rr macro
30 It's only used in one place and not very intuitive
32 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
34 fsl-dpaa2: eth: Rename a variable
36 The old name was a leftover and non-intuitive.
38 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
40 fsl-dpaa2: eth: Rearrange code
42 Rearrange the conditional statements in several functions
43 to avoid excessive indenting, with no change in functionality.
45 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
47 fsl-dpaa2: eth: Remove incorrect check
49 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
51 fsl-dpaa2: eth: Fix bug on error path
53 We were not doing a DMA unmap on the error path of dpaa2_dpni_setup.
54 Reorganize the code a bit to avoid this.
56 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
58 fsl-dpaa2: eth: Error messages cleanup
60 This commit cleans up and improves uniformity of messages on
61 error paths throughout the Ethernet driver:
63 * don't use WARN/WARN_ON/WARN_ONCE for warning messages, as
64 we don't need a stack dump
65 * give up using the DPAA2_ETH_WARN_IF_ERR custom macro
66 * ensure dev_err and netdev_err are each used where needed and
68 * remove error messages on memory allocation failures; the kernel
69 is quite capable of dumping a detailed message when that happens
70 * remove error messages on the fast path; we don't want to flood
71 the console and we already increment counters in most error cases
72 * ratelimit error messages where appropriate
74 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
76 fsl-dpaa2: eth: Fix name of ethtool counters
78 Rename counters in ethtool -S from "portal busy" to "dequeue portal
79 busy" and from "tx portal busy" to "enqueue portal busy", so it's
80 less confusing for the user.
82 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
84 fsl-dpaa2: eth: Retry DAN rearm if portal busy
86 There's a chance the data available notification rearming will
87 fail if the QBMan portal is busy. Keep retrying until portal
88 becomes available again, like we do for buffer release and
89 pull dequeue operations.
91 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
93 fsl-dpaa2: eth: Add cpu_relax() to portal busy loops
95 For several DPIO operations, we may need to repeatedly try
96 until the QBMan portal is no longer busy. Add a cpu_relax() to
97 those loops, like we were already doing when seeding buffers.
99 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
101 fsl-dpaa2: eth: Add a counter for channel pull errors
103 We no longer print an error message in this case, so add an error
104 counter so we can at least know something went wrong.
106 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
108 fsl-dpaa2: eth: Function renames
110 Attempt to provide more uniformity for the DPAA2 Ethernet
111 driver function naming conventions:
112 * major functions (ndo_ops, driver ops, ethtool, etc) all have
113 the "dpaa2_eth" prefix
114 * non-static functions also start with "dpaa2_eth"
115 * static helper functions don't get any prefix in order to avoid
117 * some functions get more intuitive and/or explicit names
118 * don't use names starting with an underscore
120 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
122 fsl-dpaa2: eth: Structure and macro renames
125 * defines of error/status bits in the frame annotation status
126 word get a "DPAA2_FAS" prefix instead of "DPAA2_ETH_FAS", as they're
127 not really specific to the ethernet driver. We may consider moving
128 these defines to a separate header file in the future
129 * DPAA2_ETH_RX_BUFFER_SIZE is renamed to DPAA2_ETH_RX_BUF_SIZE
130 to better match the naming style of other defines
131 * structure "dpaa2_eth_stats" becomes "dpaa2_eth_drv_stats" to
132 make it clear these are driver specific statistics
134 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
136 fsl-dpaa2: eth: Cosmetics
138 Various coding style fixes and other minor cosmetics,
139 with no functional impact. Also remove a couple of unused
140 defines and a structure field.
142 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
144 fsl-dpaa2: eth: Move function call
146 Move call to set_fq_affinity() from probe to setup_fqs(), as it
147 logically belongs there.
149 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@freescale.com>
151 fsl-dpaa2: eth: Comments cleanup
153 Add relevant comments where needed, remove obsolete or
156 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
158 fsl-dpaa2: eth: Remove link poll Kconfig option
160 Always try to use interrupts, but if they are not available
161 fall back to polling the link state.
163 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
165 fsl-dpaa2: eth: Remove message level
167 We were defining netif message level, but we weren't using
168 it when printing error/info messages, so remove for now.
170 Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
172 fsl-dpaa2: eth: fix compile error on 4.5 uprev
174 Signed-off-by: Stuart Yoder <stuart.yoder@nxp.com>
176 drivers/staging/fsl-dpaa2/ethernet/Kconfig | 6 -
177 .../staging/fsl-dpaa2/ethernet/dpaa2-eth-debugfs.c | 6 +-
178 drivers/staging/fsl-dpaa2/ethernet/dpaa2-eth.c | 992 ++++++++++----------
179 drivers/staging/fsl-dpaa2/ethernet/dpaa2-eth.h | 133 +--
180 drivers/staging/fsl-dpaa2/ethernet/dpaa2-ethtool.c | 226 ++---
181 5 files changed, 693 insertions(+), 670 deletions(-)
183 --- a/drivers/staging/fsl-dpaa2/ethernet/Kconfig
184 +++ b/drivers/staging/fsl-dpaa2/ethernet/Kconfig
185 @@ -16,12 +16,6 @@ menuconfig FSL_DPAA2_ETH
186 driver, using the Freescale MC bus driver.
189 -config FSL_DPAA2_ETH_LINK_POLL
190 - bool "Use polling mode for link state"
193 - Poll for detecting link state changes instead of using
196 config FSL_DPAA2_ETH_USE_ERR_QUEUE
197 bool "Enable Rx error queue"
198 --- a/drivers/staging/fsl-dpaa2/ethernet/dpaa2-eth-debugfs.c
199 +++ b/drivers/staging/fsl-dpaa2/ethernet/dpaa2-eth-debugfs.c
201 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
205 #include <linux/module.h>
206 #include <linux/debugfs.h>
207 #include "dpaa2-eth.h"
210 #define DPAA2_ETH_DBG_ROOT "dpaa2-eth"
213 static struct dentry *dpaa2_dbg_root;
215 static int dpaa2_dbg_cpu_show(struct seq_file *file, void *offset)
217 struct dpaa2_eth_priv *priv = (struct dpaa2_eth_priv *)file->private;
218 struct rtnl_link_stats64 *stats;
219 - struct dpaa2_eth_stats *extras;
220 + struct dpaa2_eth_drv_stats *extras;
223 seq_printf(file, "Per-CPU stats for %s\n", priv->net_dev->name);
224 @@ -200,7 +198,7 @@ static ssize_t dpaa2_dbg_reset_write(str
226 struct dpaa2_eth_priv *priv = file->private_data;
227 struct rtnl_link_stats64 *percpu_stats;
228 - struct dpaa2_eth_stats *percpu_extras;
229 + struct dpaa2_eth_drv_stats *percpu_extras;
230 struct dpaa2_eth_fq *fq;
231 struct dpaa2_eth_channel *ch;
233 --- a/drivers/staging/fsl-dpaa2/ethernet/dpaa2-eth.c
234 +++ b/drivers/staging/fsl-dpaa2/ethernet/dpaa2-eth.c
235 @@ -53,26 +53,14 @@ MODULE_LICENSE("Dual BSD/GPL");
236 MODULE_AUTHOR("Freescale Semiconductor, Inc");
237 MODULE_DESCRIPTION("Freescale DPAA2 Ethernet Driver");
239 -static int debug = -1;
240 -module_param(debug, int, S_IRUGO);
241 -MODULE_PARM_DESC(debug, "Module/Driver verbosity level");
243 /* Oldest DPAA2 objects version we are compatible with */
244 #define DPAA2_SUPPORTED_DPNI_VERSION 6
245 #define DPAA2_SUPPORTED_DPBP_VERSION 2
246 #define DPAA2_SUPPORTED_DPCON_VERSION 2
248 -/* Iterate through the cpumask in a round-robin fashion. */
249 -#define cpumask_rr(cpu, maskptr) \
251 - (cpu) = cpumask_next((cpu), (maskptr)); \
252 - if ((cpu) >= nr_cpu_ids) \
253 - (cpu) = cpumask_first((maskptr)); \
256 -static void dpaa2_eth_rx_csum(struct dpaa2_eth_priv *priv,
258 - struct sk_buff *skb)
259 +static void validate_rx_csum(struct dpaa2_eth_priv *priv,
261 + struct sk_buff *skb)
263 skb_checksum_none_assert(skb);
265 @@ -81,8 +69,8 @@ static void dpaa2_eth_rx_csum(struct dpa
268 /* Read checksum validation bits */
269 - if (!((fd_status & DPAA2_ETH_FAS_L3CV) &&
270 - (fd_status & DPAA2_ETH_FAS_L4CV)))
271 + if (!((fd_status & DPAA2_FAS_L3CV) &&
272 + (fd_status & DPAA2_FAS_L4CV)))
275 /* Inform the stack there's no need to compute L3/L4 csum anymore */
276 @@ -92,53 +80,55 @@ static void dpaa2_eth_rx_csum(struct dpa
277 /* Free a received FD.
278 * Not to be used for Tx conf FDs or on any other paths.
280 -static void dpaa2_eth_free_rx_fd(struct dpaa2_eth_priv *priv,
281 - const struct dpaa2_fd *fd,
283 +static void free_rx_fd(struct dpaa2_eth_priv *priv,
284 + const struct dpaa2_fd *fd,
287 struct device *dev = priv->net_dev->dev.parent;
288 dma_addr_t addr = dpaa2_fd_get_addr(fd);
289 u8 fd_format = dpaa2_fd_get_format(fd);
290 + struct dpaa2_sg_entry *sgt;
294 - if (fd_format == dpaa2_fd_sg) {
295 - struct dpaa2_sg_entry *sgt = vaddr + dpaa2_fd_get_offset(fd);
298 + /* If single buffer frame, just free the data buffer */
299 + if (fd_format == dpaa2_fd_single)
302 - for (i = 0; i < DPAA2_ETH_MAX_SG_ENTRIES; i++) {
303 - dpaa2_sg_le_to_cpu(&sgt[i]);
304 + /* For S/G frames, we first need to free all SG entries */
305 + sgt = vaddr + dpaa2_fd_get_offset(fd);
306 + for (i = 0; i < DPAA2_ETH_MAX_SG_ENTRIES; i++) {
307 + dpaa2_sg_le_to_cpu(&sgt[i]);
309 - addr = dpaa2_sg_get_addr(&sgt[i]);
310 - dma_unmap_single(dev, addr, DPAA2_ETH_RX_BUFFER_SIZE,
312 + addr = dpaa2_sg_get_addr(&sgt[i]);
313 + dma_unmap_single(dev, addr, DPAA2_ETH_RX_BUF_SIZE,
316 - sg_vaddr = phys_to_virt(addr);
317 - put_page(virt_to_head_page(sg_vaddr));
318 + sg_vaddr = phys_to_virt(addr);
319 + put_page(virt_to_head_page(sg_vaddr));
321 - if (dpaa2_sg_is_final(&sgt[i]))
324 + if (dpaa2_sg_is_final(&sgt[i]))
329 put_page(virt_to_head_page(vaddr));
332 /* Build a linear skb based on a single-buffer frame descriptor */
333 -static struct sk_buff *dpaa2_eth_build_linear_skb(struct dpaa2_eth_priv *priv,
334 - struct dpaa2_eth_channel *ch,
335 - const struct dpaa2_fd *fd,
337 +static struct sk_buff *build_linear_skb(struct dpaa2_eth_priv *priv,
338 + struct dpaa2_eth_channel *ch,
339 + const struct dpaa2_fd *fd,
342 struct sk_buff *skb = NULL;
343 u16 fd_offset = dpaa2_fd_get_offset(fd);
344 u32 fd_length = dpaa2_fd_get_len(fd);
346 - skb = build_skb(fd_vaddr, DPAA2_ETH_RX_BUFFER_SIZE +
347 + skb = build_skb(fd_vaddr, DPAA2_ETH_RX_BUF_SIZE +
348 SKB_DATA_ALIGN(sizeof(struct skb_shared_info)));
349 - if (unlikely(!skb)) {
350 - netdev_err(priv->net_dev, "build_skb() failed\n");
351 + if (unlikely(!skb))
355 skb_reserve(skb, fd_offset);
356 skb_put(skb, fd_length);
357 @@ -149,9 +139,9 @@ static struct sk_buff *dpaa2_eth_build_l
360 /* Build a non linear (fragmented) skb based on a S/G table */
361 -static struct sk_buff *dpaa2_eth_build_frag_skb(struct dpaa2_eth_priv *priv,
362 - struct dpaa2_eth_channel *ch,
363 - struct dpaa2_sg_entry *sgt)
364 +static struct sk_buff *build_frag_skb(struct dpaa2_eth_priv *priv,
365 + struct dpaa2_eth_channel *ch,
366 + struct dpaa2_sg_entry *sgt)
368 struct sk_buff *skb = NULL;
369 struct device *dev = priv->net_dev->dev.parent;
370 @@ -168,66 +158,57 @@ static struct sk_buff *dpaa2_eth_build_f
372 dpaa2_sg_le_to_cpu(sge);
374 - /* We don't support anything else yet! */
375 - if (unlikely(dpaa2_sg_get_format(sge) != dpaa2_sg_single)) {
376 - dev_warn_once(dev, "Unsupported S/G entry format: %d\n",
377 - dpaa2_sg_get_format(sge));
380 + /* NOTE: We only support SG entries in dpaa2_sg_single format,
381 + * but this is the only format we may receive from HW anyway
384 - /* Get the address, offset and length from the S/G entry */
385 + /* Get the address and length from the S/G entry */
386 sg_addr = dpaa2_sg_get_addr(sge);
387 - dma_unmap_single(dev, sg_addr, DPAA2_ETH_RX_BUFFER_SIZE,
388 + dma_unmap_single(dev, sg_addr, DPAA2_ETH_RX_BUF_SIZE,
390 - if (unlikely(dma_mapping_error(dev, sg_addr))) {
391 - netdev_err(priv->net_dev, "DMA unmap failed\n");
395 sg_vaddr = phys_to_virt(sg_addr);
396 sg_length = dpaa2_sg_get_len(sge);
399 /* We build the skb around the first data buffer */
400 - skb = build_skb(sg_vaddr, DPAA2_ETH_RX_BUFFER_SIZE +
401 + skb = build_skb(sg_vaddr, DPAA2_ETH_RX_BUF_SIZE +
402 SKB_DATA_ALIGN(sizeof(struct skb_shared_info)));
403 - if (unlikely(!skb)) {
404 - netdev_err(priv->net_dev, "build_skb failed\n");
405 + if (unlikely(!skb))
409 sg_offset = dpaa2_sg_get_offset(sge);
410 skb_reserve(skb, sg_offset);
411 skb_put(skb, sg_length);
413 - /* Subsequent data in SGEntries are stored at
414 - * offset 0 in their buffers, we don't need to
415 - * compute sg_offset.
417 - WARN_ONCE(dpaa2_sg_get_offset(sge) != 0,
418 - "Non-zero offset in SGE[%d]!\n", i);
420 /* Rest of the data buffers are stored as skb frags */
421 page = virt_to_page(sg_vaddr);
422 head_page = virt_to_head_page(sg_vaddr);
424 - /* Offset in page (which may be compound) */
425 + /* Offset in page (which may be compound).
426 + * Data in subsequent SG entries is stored from the
427 + * beginning of the buffer, so we don't need to add the
430 page_offset = ((unsigned long)sg_vaddr &
432 (page_address(page) - page_address(head_page));
434 skb_add_rx_frag(skb, i - 1, head_page, page_offset,
435 - sg_length, DPAA2_ETH_RX_BUFFER_SIZE);
436 + sg_length, DPAA2_ETH_RX_BUF_SIZE);
439 if (dpaa2_sg_is_final(sge))
443 - /* Count all data buffers + sgt buffer */
444 + /* Count all data buffers + SG table buffer */
445 ch->buf_count -= i + 2;
450 +/* Main Rx frame processing routine */
451 static void dpaa2_eth_rx(struct dpaa2_eth_priv *priv,
452 struct dpaa2_eth_channel *ch,
453 const struct dpaa2_fd *fd,
454 @@ -238,7 +219,7 @@ static void dpaa2_eth_rx(struct dpaa2_et
457 struct rtnl_link_stats64 *percpu_stats;
458 - struct dpaa2_eth_stats *percpu_extras;
459 + struct dpaa2_eth_drv_stats *percpu_extras;
460 struct device *dev = priv->net_dev->dev.parent;
461 struct dpaa2_fas *fas;
463 @@ -246,7 +227,7 @@ static void dpaa2_eth_rx(struct dpaa2_et
465 trace_dpaa2_rx_fd(priv->net_dev, fd);
467 - dma_unmap_single(dev, addr, DPAA2_ETH_RX_BUFFER_SIZE, DMA_FROM_DEVICE);
468 + dma_unmap_single(dev, addr, DPAA2_ETH_RX_BUF_SIZE, DMA_FROM_DEVICE);
469 vaddr = phys_to_virt(addr);
471 prefetch(vaddr + priv->buf_layout.private_data_size);
472 @@ -256,32 +237,30 @@ static void dpaa2_eth_rx(struct dpaa2_et
473 percpu_extras = this_cpu_ptr(priv->percpu_extras);
475 if (fd_format == dpaa2_fd_single) {
476 - skb = dpaa2_eth_build_linear_skb(priv, ch, fd, vaddr);
477 + skb = build_linear_skb(priv, ch, fd, vaddr);
478 } else if (fd_format == dpaa2_fd_sg) {
479 struct dpaa2_sg_entry *sgt =
480 vaddr + dpaa2_fd_get_offset(fd);
481 - skb = dpaa2_eth_build_frag_skb(priv, ch, sgt);
482 + skb = build_frag_skb(priv, ch, sgt);
483 put_page(virt_to_head_page(vaddr));
484 percpu_extras->rx_sg_frames++;
485 percpu_extras->rx_sg_bytes += dpaa2_fd_get_len(fd);
487 /* We don't support any other format */
488 - netdev_err(priv->net_dev, "Received invalid frame format\n");
489 goto err_frame_format;
492 - if (unlikely(!skb)) {
493 - dev_err_once(dev, "error building skb\n");
494 + if (unlikely(!skb))
500 + /* Get the timestamp value */
501 if (priv->ts_rx_en) {
502 struct skb_shared_hwtstamps *shhwtstamps = skb_hwtstamps(skb);
503 - u64 *ns = (u64 *) (vaddr +
504 - priv->buf_layout.private_data_size +
505 - sizeof(struct dpaa2_fas));
506 + u64 *ns = (u64 *)(vaddr +
507 + priv->buf_layout.private_data_size +
508 + sizeof(struct dpaa2_fas));
510 *ns = DPAA2_PTP_NOMINAL_FREQ_PERIOD_NS * (*ns);
511 memset(shhwtstamps, 0, sizeof(*shhwtstamps));
512 @@ -293,7 +272,7 @@ static void dpaa2_eth_rx(struct dpaa2_et
513 fas = (struct dpaa2_fas *)
514 (vaddr + priv->buf_layout.private_data_size);
515 status = le32_to_cpu(fas->status);
516 - dpaa2_eth_rx_csum(priv, status, skb);
517 + validate_rx_csum(priv, status, skb);
520 skb->protocol = eth_type_trans(skb, priv->net_dev);
521 @@ -309,11 +288,14 @@ static void dpaa2_eth_rx(struct dpaa2_et
525 - dpaa2_eth_free_rx_fd(priv, fd, vaddr);
526 + free_rx_fd(priv, fd, vaddr);
527 percpu_stats->rx_dropped++;
530 #ifdef CONFIG_FSL_DPAA2_ETH_USE_ERR_QUEUE
531 +/* Processing of Rx frames received on the error FQ
532 + * We check and print the error bits and then free the frame
534 static void dpaa2_eth_rx_err(struct dpaa2_eth_priv *priv,
535 struct dpaa2_eth_channel *ch,
536 const struct dpaa2_fd *fd,
537 @@ -326,21 +308,18 @@ static void dpaa2_eth_rx_err(struct dpaa
538 struct dpaa2_fas *fas;
541 - dma_unmap_single(dev, addr, DPAA2_ETH_RX_BUFFER_SIZE, DMA_FROM_DEVICE);
542 + dma_unmap_single(dev, addr, DPAA2_ETH_RX_BUF_SIZE, DMA_FROM_DEVICE);
543 vaddr = phys_to_virt(addr);
545 if (fd->simple.frc & DPAA2_FD_FRC_FASV) {
546 fas = (struct dpaa2_fas *)
547 (vaddr + priv->buf_layout.private_data_size);
548 status = le32_to_cpu(fas->status);
550 - /* All frames received on this queue should have at least
551 - * one of the Rx error bits set */
552 - WARN_ON_ONCE((status & DPAA2_ETH_RX_ERR_MASK) == 0);
553 - netdev_dbg(priv->net_dev, "Rx frame error: 0x%08x\n",
554 - status & DPAA2_ETH_RX_ERR_MASK);
555 + if (net_ratelimit())
556 + netdev_warn(priv->net_dev, "Rx frame error: 0x%08x\n",
557 + status & DPAA2_ETH_RX_ERR_MASK);
559 - dpaa2_eth_free_rx_fd(priv, fd, vaddr);
560 + free_rx_fd(priv, fd, vaddr);
562 percpu_stats = this_cpu_ptr(priv->percpu_stats);
563 percpu_stats->rx_errors++;
564 @@ -353,7 +332,7 @@ static void dpaa2_eth_rx_err(struct dpaa
566 * Observance of NAPI budget is not our concern, leaving that to the caller.
568 -static int dpaa2_eth_store_consume(struct dpaa2_eth_channel *ch)
569 +static int consume_frames(struct dpaa2_eth_channel *ch)
571 struct dpaa2_eth_priv *priv = ch->priv;
572 struct dpaa2_eth_fq *fq;
573 @@ -365,20 +344,14 @@ static int dpaa2_eth_store_consume(struc
575 dq = dpaa2_io_store_next(ch->store, &is_last);
577 - if (unlikely(!is_last)) {
578 - netdev_dbg(priv->net_dev,
579 - "Channel %d reqturned no valid frames\n",
581 - /* MUST retry until we get some sort of
582 - * valid response token (be it "empty dequeue"
583 - * or a valid frame).
588 + /* If we're here, we *must* have placed a
589 + * volatile dequeue comnmand, so keep reading through
590 + * the store until we get some sort of valid response
591 + * token (either a valid frame or an "empty dequeue")
596 - /* Obtain FD and process it */
597 fd = dpaa2_dq_fd(dq);
598 fq = (struct dpaa2_eth_fq *)dpaa2_dq_fqd_ctx(dq);
600 @@ -390,9 +363,10 @@ static int dpaa2_eth_store_consume(struc
604 -static int dpaa2_eth_build_sg_fd(struct dpaa2_eth_priv *priv,
605 - struct sk_buff *skb,
606 - struct dpaa2_fd *fd)
607 +/* Create a frame descriptor based on a fragmented skb */
608 +static int build_sg_fd(struct dpaa2_eth_priv *priv,
609 + struct sk_buff *skb,
610 + struct dpaa2_fd *fd)
612 struct device *dev = priv->net_dev->dev.parent;
613 void *sgt_buf = NULL;
614 @@ -404,14 +378,16 @@ static int dpaa2_eth_build_sg_fd(struct
615 struct scatterlist *scl, *crt_scl;
618 - struct dpaa2_eth_swa *bps;
619 + struct dpaa2_eth_swa *swa;
621 /* Create and map scatterlist.
622 * We don't advertise NETIF_F_FRAGLIST, so skb_to_sgvec() will not have
623 * to go beyond nr_frags+1.
624 * Note: We don't support chained scatterlists
626 - WARN_ON(PAGE_SIZE / sizeof(struct scatterlist) < nr_frags + 1);
627 + if (unlikely(PAGE_SIZE / sizeof(struct scatterlist) < nr_frags + 1))
630 scl = kcalloc(nr_frags + 1, sizeof(struct scatterlist), GFP_ATOMIC);
633 @@ -420,7 +396,6 @@ static int dpaa2_eth_build_sg_fd(struct
634 num_sg = skb_to_sgvec(skb, scl, 0, skb->len);
635 num_dma_bufs = dma_map_sg(dev, scl, num_sg, DMA_TO_DEVICE);
636 if (unlikely(!num_dma_bufs)) {
637 - netdev_err(priv->net_dev, "dma_map_sg() error\n");
639 goto dma_map_sg_failed;
641 @@ -430,7 +405,6 @@ static int dpaa2_eth_build_sg_fd(struct
642 sizeof(struct dpaa2_sg_entry) * (1 + num_dma_bufs);
643 sgt_buf = kzalloc(sgt_buf_size + DPAA2_ETH_TX_BUF_ALIGN, GFP_ATOMIC);
644 if (unlikely(!sgt_buf)) {
645 - netdev_err(priv->net_dev, "failed to allocate SGT buffer\n");
647 goto sgt_buf_alloc_failed;
649 @@ -462,19 +436,19 @@ static int dpaa2_eth_build_sg_fd(struct
650 * Fit the scatterlist and the number of buffers alongside the
651 * skb backpointer in the SWA. We'll need all of them on Tx Conf.
653 - bps = (struct dpaa2_eth_swa *)sgt_buf;
656 - bps->num_sg = num_sg;
657 - bps->num_dma_bufs = num_dma_bufs;
658 + swa = (struct dpaa2_eth_swa *)sgt_buf;
661 + swa->num_sg = num_sg;
662 + swa->num_dma_bufs = num_dma_bufs;
664 + /* Hardware expects the SG table to be in little endian format */
665 for (j = 0; j < i; j++)
666 dpaa2_sg_cpu_to_le(&sgt[j]);
668 /* Separately map the SGT buffer */
669 addr = dma_map_single(dev, sgt_buf, sgt_buf_size, DMA_TO_DEVICE);
670 if (unlikely(dma_mapping_error(dev, addr))) {
671 - netdev_err(priv->net_dev, "dma_map_single() failed\n");
673 goto dma_map_single_failed;
675 @@ -484,7 +458,7 @@ static int dpaa2_eth_build_sg_fd(struct
676 dpaa2_fd_set_len(fd, skb->len);
678 fd->simple.ctrl = DPAA2_FD_CTRL_ASAL | DPAA2_FD_CTRL_PTA |
679 - DPAA2_FD_CTRL_PTV1;
680 + DPAA2_FD_CTRL_PTV1;
684 @@ -497,9 +471,10 @@ dma_map_sg_failed:
688 -static int dpaa2_eth_build_single_fd(struct dpaa2_eth_priv *priv,
689 - struct sk_buff *skb,
690 - struct dpaa2_fd *fd)
691 +/* Create a frame descriptor based on a linear skb */
692 +static int build_single_fd(struct dpaa2_eth_priv *priv,
693 + struct sk_buff *skb,
694 + struct dpaa2_fd *fd)
696 struct device *dev = priv->net_dev->dev.parent;
698 @@ -524,14 +499,11 @@ static int dpaa2_eth_build_single_fd(str
699 skbh = (struct sk_buff **)buffer_start;
702 - addr = dma_map_single(dev,
704 + addr = dma_map_single(dev, buffer_start,
705 skb_tail_pointer(skb) - buffer_start,
707 - if (unlikely(dma_mapping_error(dev, addr))) {
708 - dev_err(dev, "dma_map_single() failed\n");
711 + if (unlikely(dma_mapping_error(dev, addr)))
714 dpaa2_fd_set_addr(fd, addr);
715 dpaa2_fd_set_offset(fd, (u16)(skb->data - buffer_start));
716 @@ -539,21 +511,23 @@ static int dpaa2_eth_build_single_fd(str
717 dpaa2_fd_set_format(fd, dpaa2_fd_single);
719 fd->simple.ctrl = DPAA2_FD_CTRL_ASAL | DPAA2_FD_CTRL_PTA |
720 - DPAA2_FD_CTRL_PTV1;
721 + DPAA2_FD_CTRL_PTV1;
726 -/* DMA-unmap and free FD and possibly SGT buffer allocated on Tx. The skb
727 +/* FD freeing routine on the Tx path
729 + * DMA-unmap and free FD and possibly SGT buffer allocated on Tx. The skb
730 * back-pointed to is also freed.
731 * This can be called either from dpaa2_eth_tx_conf() or on the error path of
733 * Optionally, return the frame annotation status word (FAS), which needs
734 * to be checked if we're on the confirmation path.
736 -static void dpaa2_eth_free_fd(const struct dpaa2_eth_priv *priv,
737 - const struct dpaa2_fd *fd,
739 +static void free_tx_fd(const struct dpaa2_eth_priv *priv,
740 + const struct dpaa2_fd *fd,
743 struct device *dev = priv->net_dev->dev.parent;
745 @@ -562,7 +536,7 @@ static void dpaa2_eth_free_fd(const stru
747 struct scatterlist *scl;
748 int num_sg, num_dma_bufs;
749 - struct dpaa2_eth_swa *bps;
750 + struct dpaa2_eth_swa *swa;
752 struct dpaa2_fas *fas;
754 @@ -580,11 +554,11 @@ static void dpaa2_eth_free_fd(const stru
755 skb_tail_pointer(skb) - buffer_start,
758 - bps = (struct dpaa2_eth_swa *)skbh;
761 - num_sg = bps->num_sg;
762 - num_dma_bufs = bps->num_dma_bufs;
763 + swa = (struct dpaa2_eth_swa *)skbh;
766 + num_sg = swa->num_sg;
767 + num_dma_bufs = swa->num_dma_bufs;
769 /* Unmap the scatterlist */
770 dma_unmap_sg(dev, scl, num_sg, DMA_TO_DEVICE);
771 @@ -596,6 +570,7 @@ static void dpaa2_eth_free_fd(const stru
772 dma_unmap_single(dev, fd_addr, unmap_size, DMA_TO_DEVICE);
775 + /* Get the timestamp value */
776 if (priv->ts_tx_en && skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) {
777 struct skb_shared_hwtstamps shhwtstamps;
779 @@ -610,8 +585,9 @@ static void dpaa2_eth_free_fd(const stru
780 skb_tstamp_tx(skb, &shhwtstamps);
783 - /* Check the status from the Frame Annotation after we unmap the first
784 - * buffer but before we free it.
785 + /* Read the status from the Frame Annotation after we unmap the first
786 + * buffer but before we free it. The caller function is responsible
787 + * for checking the status value.
789 if (status && (fd->simple.frc & DPAA2_FD_FRC_FASV)) {
790 fas = (struct dpaa2_fas *)
791 @@ -632,24 +608,16 @@ static int dpaa2_eth_tx(struct sk_buff *
792 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
794 struct rtnl_link_stats64 *percpu_stats;
795 - struct dpaa2_eth_stats *percpu_extras;
796 + struct dpaa2_eth_drv_stats *percpu_extras;
797 + u16 queue_mapping, flow_id;
799 - /* TxConf FQ selection primarily based on cpu affinity; this is
800 - * non-migratable context, so it's safe to call smp_processor_id().
802 - u16 queue_mapping = smp_processor_id() % priv->dpni_attrs.max_senders;
804 percpu_stats = this_cpu_ptr(priv->percpu_stats);
805 percpu_extras = this_cpu_ptr(priv->percpu_extras);
807 - /* Setup the FD fields */
808 - memset(&fd, 0, sizeof(fd));
810 if (unlikely(skb_headroom(skb) < DPAA2_ETH_NEEDED_HEADROOM(priv))) {
813 - dev_info_once(net_dev->dev.parent,
814 - "skb headroom too small, must realloc.\n");
815 ns = skb_realloc_headroom(skb, DPAA2_ETH_NEEDED_HEADROOM(priv));
817 percpu_stats->tx_dropped++;
818 @@ -664,18 +632,20 @@ static int dpaa2_eth_tx(struct sk_buff *
820 skb = skb_unshare(skb, GFP_ATOMIC);
821 if (unlikely(!skb)) {
822 - netdev_err(net_dev, "Out of memory for skb_unshare()");
823 /* skb_unshare() has already freed the skb */
824 percpu_stats->tx_dropped++;
828 + /* Setup the FD fields */
829 + memset(&fd, 0, sizeof(fd));
831 if (skb_is_nonlinear(skb)) {
832 - err = dpaa2_eth_build_sg_fd(priv, skb, &fd);
833 + err = build_sg_fd(priv, skb, &fd);
834 percpu_extras->tx_sg_frames++;
835 percpu_extras->tx_sg_bytes += skb->len;
837 - err = dpaa2_eth_build_single_fd(priv, skb, &fd);
838 + err = build_single_fd(priv, skb, &fd);
842 @@ -686,19 +656,22 @@ static int dpaa2_eth_tx(struct sk_buff *
844 trace_dpaa2_tx_fd(net_dev, &fd);
846 + /* TxConf FQ selection primarily based on cpu affinity; this is
847 + * non-migratable context, so it's safe to call smp_processor_id().
849 + queue_mapping = smp_processor_id() % priv->dpni_attrs.max_senders;
850 + flow_id = priv->fq[queue_mapping].flowid;
851 for (i = 0; i < (DPAA2_ETH_MAX_TX_QUEUES << 1); i++) {
852 err = dpaa2_io_service_enqueue_qd(NULL, priv->tx_qdid, 0,
853 - priv->fq[queue_mapping].flowid,
859 percpu_extras->tx_portal_busy += i;
860 if (unlikely(err < 0)) {
861 - netdev_dbg(net_dev, "error enqueueing Tx frame\n");
862 percpu_stats->tx_errors++;
863 /* Clean up everything, including freeing the skb */
864 - dpaa2_eth_free_fd(priv, &fd, NULL);
865 + free_tx_fd(priv, &fd, NULL);
867 percpu_stats->tx_packets++;
868 percpu_stats->tx_bytes += skb->len;
869 @@ -713,13 +686,14 @@ err_alloc_headroom:
873 +/* Tx confirmation frame processing routine */
874 static void dpaa2_eth_tx_conf(struct dpaa2_eth_priv *priv,
875 struct dpaa2_eth_channel *ch,
876 const struct dpaa2_fd *fd,
877 struct napi_struct *napi __always_unused)
879 struct rtnl_link_stats64 *percpu_stats;
880 - struct dpaa2_eth_stats *percpu_extras;
881 + struct dpaa2_eth_drv_stats *percpu_extras;
885 @@ -729,18 +703,16 @@ static void dpaa2_eth_tx_conf(struct dpa
886 percpu_extras->tx_conf_frames++;
887 percpu_extras->tx_conf_bytes += dpaa2_fd_get_len(fd);
889 - dpaa2_eth_free_fd(priv, fd, &status);
890 + free_tx_fd(priv, fd, &status);
892 if (unlikely(status & DPAA2_ETH_TXCONF_ERR_MASK)) {
893 - netdev_err(priv->net_dev, "TxConf frame error(s): 0x%08x\n",
894 - status & DPAA2_ETH_TXCONF_ERR_MASK);
895 percpu_stats = this_cpu_ptr(priv->percpu_stats);
896 /* Tx-conf logically pertains to the egress path. */
897 percpu_stats->tx_errors++;
901 -static int dpaa2_eth_set_rx_csum(struct dpaa2_eth_priv *priv, bool enable)
902 +static int set_rx_csum(struct dpaa2_eth_priv *priv, bool enable)
906 @@ -763,7 +735,7 @@ static int dpaa2_eth_set_rx_csum(struct
910 -static int dpaa2_eth_set_tx_csum(struct dpaa2_eth_priv *priv, bool enable)
911 +static int set_tx_csum(struct dpaa2_eth_priv *priv, bool enable)
913 struct dpaa2_eth_fq *fq;
914 struct dpni_tx_flow_cfg tx_flow_cfg;
915 @@ -793,37 +765,38 @@ static int dpaa2_eth_set_tx_csum(struct
919 -static int dpaa2_bp_add_7(struct dpaa2_eth_priv *priv, u16 bpid)
920 +/* Perform a single release command to add buffers
921 + * to the specified buffer pool
923 +static int add_bufs(struct dpaa2_eth_priv *priv, u16 bpid)
925 struct device *dev = priv->net_dev->dev.parent;
927 + u64 buf_array[DPAA2_ETH_BUFS_PER_CMD];
932 - for (i = 0; i < 7; i++) {
933 + for (i = 0; i < DPAA2_ETH_BUFS_PER_CMD; i++) {
934 /* Allocate buffer visible to WRIOP + skb shared info +
937 buf = napi_alloc_frag(DPAA2_ETH_BUF_RAW_SIZE);
938 - if (unlikely(!buf)) {
939 - dev_err(dev, "buffer allocation failed\n");
940 + if (unlikely(!buf))
944 buf = PTR_ALIGN(buf, DPAA2_ETH_RX_BUF_ALIGN);
946 - addr = dma_map_single(dev, buf, DPAA2_ETH_RX_BUFFER_SIZE,
947 + addr = dma_map_single(dev, buf, DPAA2_ETH_RX_BUF_SIZE,
949 - if (unlikely(dma_mapping_error(dev, addr))) {
950 - dev_err(dev, "dma_map_single() failed\n");
951 + if (unlikely(dma_mapping_error(dev, addr)))
958 trace_dpaa2_eth_buf_seed(priv->net_dev,
959 buf, DPAA2_ETH_BUF_RAW_SIZE,
960 - addr, DPAA2_ETH_RX_BUFFER_SIZE,
961 + addr, DPAA2_ETH_RX_BUF_SIZE,
965 @@ -850,59 +823,57 @@ err_alloc:
969 -static int dpaa2_dpbp_seed(struct dpaa2_eth_priv *priv, u16 bpid)
970 +static int seed_pool(struct dpaa2_eth_priv *priv, u16 bpid)
975 /* This is the lazy seeding of Rx buffer pools.
976 - * dpaa2_bp_add_7() is also used on the Rx hotpath and calls
977 + * dpaa2_add_bufs() is also used on the Rx hotpath and calls
978 * napi_alloc_frag(). The trouble with that is that it in turn ends up
979 * calling this_cpu_ptr(), which mandates execution in atomic context.
980 * Rather than splitting up the code, do a one-off preempt disable.
983 for (j = 0; j < priv->num_channels; j++) {
984 - for (i = 0; i < DPAA2_ETH_NUM_BUFS; i += 7) {
985 - new_count = dpaa2_bp_add_7(priv, bpid);
986 + for (i = 0; i < DPAA2_ETH_NUM_BUFS;
987 + i += DPAA2_ETH_BUFS_PER_CMD) {
988 + new_count = add_bufs(priv, bpid);
989 priv->channel[j]->buf_count += new_count;
991 - if (new_count < 7) {
992 + if (new_count < DPAA2_ETH_BUFS_PER_CMD) {
994 - goto out_of_memory;
1008 * Drain the specified number of buffers from the DPNI's private buffer pool.
1009 - * @count must not exceeed 7
1010 + * @count must not exceeed DPAA2_ETH_BUFS_PER_CMD
1012 -static void dpaa2_dpbp_drain_cnt(struct dpaa2_eth_priv *priv, int count)
1013 +static void drain_bufs(struct dpaa2_eth_priv *priv, int count)
1015 struct device *dev = priv->net_dev->dev.parent;
1017 + u64 buf_array[DPAA2_ETH_BUFS_PER_CMD];
1022 ret = dpaa2_io_service_acquire(NULL, priv->dpbp_attrs.bpid,
1023 - buf_array, count);
1024 + buf_array, count);
1026 - pr_err("dpaa2_io_service_acquire() failed\n");
1027 + netdev_err(priv->net_dev, "dpaa2_io_service_acquire() failed\n");
1030 for (i = 0; i < ret; i++) {
1031 /* Same logic as on regular Rx path */
1032 dma_unmap_single(dev, buf_array[i],
1033 - DPAA2_ETH_RX_BUFFER_SIZE,
1034 + DPAA2_ETH_RX_BUF_SIZE,
1036 vaddr = phys_to_virt(buf_array[i]);
1037 put_page(virt_to_head_page(vaddr));
1038 @@ -910,12 +881,12 @@ static void dpaa2_dpbp_drain_cnt(struct
1042 -static void __dpaa2_dpbp_free(struct dpaa2_eth_priv *priv)
1043 +static void drain_pool(struct dpaa2_eth_priv *priv)
1047 - dpaa2_dpbp_drain_cnt(priv, 7);
1048 - dpaa2_dpbp_drain_cnt(priv, 1);
1049 + drain_bufs(priv, DPAA2_ETH_BUFS_PER_CMD);
1050 + drain_bufs(priv, 1);
1052 for (i = 0; i < priv->num_channels; i++)
1053 priv->channel[i]->buf_count = 0;
1054 @@ -924,50 +895,55 @@ static void __dpaa2_dpbp_free(struct dpa
1055 /* Function is called from softirq context only, so we don't need to guard
1056 * the access to percpu count
1058 -static int dpaa2_dpbp_refill(struct dpaa2_eth_priv *priv,
1059 - struct dpaa2_eth_channel *ch,
1061 +static int refill_pool(struct dpaa2_eth_priv *priv,
1062 + struct dpaa2_eth_channel *ch,
1068 - if (unlikely(ch->buf_count < DPAA2_ETH_REFILL_THRESH)) {
1070 - new_count = dpaa2_bp_add_7(priv, bpid);
1071 - if (unlikely(!new_count)) {
1072 - /* Out of memory; abort for now, we'll
1077 - ch->buf_count += new_count;
1078 - } while (ch->buf_count < DPAA2_ETH_NUM_BUFS);
1079 + if (likely(ch->buf_count >= DPAA2_ETH_REFILL_THRESH))
1082 - if (unlikely(ch->buf_count < DPAA2_ETH_NUM_BUFS))
1086 + new_count = add_bufs(priv, bpid);
1087 + if (unlikely(!new_count)) {
1088 + /* Out of memory; abort for now, we'll try later on */
1091 + ch->buf_count += new_count;
1092 + } while (ch->buf_count < DPAA2_ETH_NUM_BUFS);
1095 + if (unlikely(ch->buf_count < DPAA2_ETH_NUM_BUFS))
1101 -static int __dpaa2_eth_pull_channel(struct dpaa2_eth_channel *ch)
1102 +static int pull_channel(struct dpaa2_eth_channel *ch)
1106 - struct dpaa2_eth_priv *priv = ch->priv;
1108 /* Retry while portal is busy */
1110 err = dpaa2_io_service_pull_channel(NULL, ch->ch_id, ch->store);
1113 } while (err == -EBUSY);
1114 - if (unlikely(err))
1115 - netdev_err(priv->net_dev, "dpaa2_io_service_pull err %d", err);
1117 ch->stats.dequeue_portal_busy += dequeues;
1118 + if (unlikely(err))
1119 + ch->stats.pull_err++;
1124 +/* NAPI poll routine
1126 + * Frames are dequeued from the QMan channel associated with this NAPI context.
1127 + * Rx, Tx confirmation and (if configured) Rx error frames all count
1128 + * towards the NAPI budget.
1130 static int dpaa2_eth_poll(struct napi_struct *napi, int budget)
1132 struct dpaa2_eth_channel *ch;
1133 @@ -978,32 +954,32 @@ static int dpaa2_eth_poll(struct napi_st
1134 ch = container_of(napi, struct dpaa2_eth_channel, napi);
1137 - __dpaa2_eth_pull_channel(ch);
1138 + while (cleaned < budget) {
1139 + err = pull_channel(ch);
1140 + if (unlikely(err))
1144 /* Refill pool if appropriate */
1145 - dpaa2_dpbp_refill(priv, ch, priv->dpbp_attrs.bpid);
1146 + refill_pool(priv, ch, priv->dpbp_attrs.bpid);
1148 - store_cleaned = dpaa2_eth_store_consume(ch);
1149 + store_cleaned = consume_frames(ch);
1150 cleaned += store_cleaned;
1152 + /* If we have enough budget left for a full store,
1153 + * try a new pull dequeue, otherwise we're done here
1155 if (store_cleaned == 0 ||
1156 cleaned > budget - DPAA2_ETH_STORE_SIZE)
1159 - /* Try to dequeue some more */
1160 - err = __dpaa2_eth_pull_channel(ch);
1161 - if (unlikely(err))
1166 if (cleaned < budget) {
1167 napi_complete_done(napi, cleaned);
1168 - err = dpaa2_io_service_rearm(NULL, &ch->nctx);
1169 - if (unlikely(err))
1170 - netdev_err(priv->net_dev,
1171 - "Notif rearm failed for channel %d\n",
1173 + /* Re-enable data available notifications */
1175 + err = dpaa2_io_service_rearm(NULL, &ch->nctx);
1177 + } while (err == -EBUSY);
1180 ch->stats.frames += cleaned;
1181 @@ -1011,7 +987,7 @@ static int dpaa2_eth_poll(struct napi_st
1185 -static void dpaa2_eth_napi_enable(struct dpaa2_eth_priv *priv)
1186 +static void enable_ch_napi(struct dpaa2_eth_priv *priv)
1188 struct dpaa2_eth_channel *ch;
1190 @@ -1022,7 +998,7 @@ static void dpaa2_eth_napi_enable(struct
1194 -static void dpaa2_eth_napi_disable(struct dpaa2_eth_priv *priv)
1195 +static void disable_ch_napi(struct dpaa2_eth_priv *priv)
1197 struct dpaa2_eth_channel *ch;
1199 @@ -1033,7 +1009,7 @@ static void dpaa2_eth_napi_disable(struc
1203 -static int dpaa2_link_state_update(struct dpaa2_eth_priv *priv)
1204 +static int link_state_update(struct dpaa2_eth_priv *priv)
1206 struct dpni_link_state state;
1208 @@ -1069,7 +1045,7 @@ static int dpaa2_eth_open(struct net_dev
1209 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
1212 - err = dpaa2_dpbp_seed(priv, priv->dpbp_attrs.bpid);
1213 + err = seed_pool(priv, priv->dpbp_attrs.bpid);
1215 /* Not much to do; the buffer pool, though not filled up,
1216 * may still contain some buffers which would enable us
1217 @@ -1084,7 +1060,7 @@ static int dpaa2_eth_open(struct net_dev
1218 * immediately after dpni_enable();
1220 netif_tx_stop_all_queues(net_dev);
1221 - dpaa2_eth_napi_enable(priv);
1222 + enable_ch_napi(priv);
1223 /* Also, explicitly set carrier off, otherwise netif_carrier_ok() will
1224 * return true and cause 'ip link show' to report the LOWER_UP flag,
1225 * even though the link notification wasn't even received.
1226 @@ -1093,16 +1069,16 @@ static int dpaa2_eth_open(struct net_dev
1228 err = dpni_enable(priv->mc_io, 0, priv->mc_token);
1230 - dev_err(net_dev->dev.parent, "dpni_enable() failed\n");
1231 + netdev_err(net_dev, "dpni_enable() failed\n");
1235 /* If the DPMAC object has already processed the link up interrupt,
1236 * we have to learn the link state ourselves.
1238 - err = dpaa2_link_state_update(priv);
1239 + err = link_state_update(priv);
1241 - dev_err(net_dev->dev.parent, "Can't update link state\n");
1242 + netdev_err(net_dev, "Can't update link state\n");
1243 goto link_state_err;
1246 @@ -1110,26 +1086,84 @@ static int dpaa2_eth_open(struct net_dev
1250 - dpaa2_eth_napi_disable(priv);
1251 - __dpaa2_dpbp_free(priv);
1252 + disable_ch_napi(priv);
1257 +/* The DPIO store must be empty when we call this,
1258 + * at the end of every NAPI cycle.
1260 +static u32 drain_channel(struct dpaa2_eth_priv *priv,
1261 + struct dpaa2_eth_channel *ch)
1263 + u32 drained = 0, total = 0;
1267 + drained = consume_frames(ch);
1269 + } while (drained);
1274 +static u32 drain_ingress_frames(struct dpaa2_eth_priv *priv)
1276 + struct dpaa2_eth_channel *ch;
1280 + for (i = 0; i < priv->num_channels; i++) {
1281 + ch = priv->channel[i];
1282 + drained += drain_channel(priv, ch);
1288 static int dpaa2_eth_stop(struct net_device *net_dev)
1290 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
1295 - /* Stop Tx and Rx traffic */
1296 netif_tx_stop_all_queues(net_dev);
1297 netif_carrier_off(net_dev);
1298 - dpni_disable(priv->mc_io, 0, priv->mc_token);
1301 + /* Loop while dpni_disable() attempts to drain the egress FQs
1302 + * and confirm them back to us.
1305 + dpni_disable(priv->mc_io, 0, priv->mc_token);
1306 + dpni_is_enabled(priv->mc_io, 0, priv->mc_token, &dpni_enabled);
1308 + /* Allow the MC some slack */
1310 + } while (dpni_enabled && --retries);
1312 + netdev_warn(net_dev, "Retry count exceeded disabling DPNI\n");
1313 + /* Must go on and disable NAPI nonetheless, so we don't crash at
1314 + * the next "ifconfig up"
1318 - dpaa2_eth_napi_disable(priv);
1320 + /* Wait for NAPI to complete on every core and disable it.
1321 + * In particular, this will also prevent NAPI from being rescheduled if
1322 + * a new CDAN is serviced, effectively discarding the CDAN. We therefore
1323 + * don't even need to disarm the channels, except perhaps for the case
1324 + * of a huge coalescing value.
1326 + disable_ch_napi(priv);
1328 + /* Manually drain the Rx and TxConf queues */
1329 + drained = drain_ingress_frames(priv);
1331 + netdev_dbg(net_dev, "Drained %d frames.\n", drained);
1333 - __dpaa2_dpbp_free(priv);
1334 + /* Empty the buffer pool */
1339 @@ -1138,7 +1172,7 @@ static int dpaa2_eth_init(struct net_dev
1342 u64 not_supported = 0;
1343 - const struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
1344 + struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
1345 u32 options = priv->dpni_attrs.options;
1347 /* Capabilities listing */
1348 @@ -1230,7 +1264,7 @@ static int dpaa2_eth_change_mtu(struct n
1349 err = dpni_set_max_frame_length(priv->mc_io, 0, priv->mc_token,
1350 (u16)DPAA2_ETH_L2_MAX_FRM(mtu));
1352 - netdev_err(net_dev, "dpni_set_mfl() failed\n");
1353 + netdev_err(net_dev, "dpni_set_max_frame_length() failed\n");
1357 @@ -1238,18 +1272,11 @@ static int dpaa2_eth_change_mtu(struct n
1361 -/* Convenience macro to make code littered with error checking more readable */
1362 -#define DPAA2_ETH_WARN_IF_ERR(err, netdevp, format, ...) \
1365 - netdev_warn(netdevp, format, ##__VA_ARGS__); \
1368 /* Copy mac unicast addresses from @net_dev to @priv.
1369 * Its sole purpose is to make dpaa2_eth_set_rx_mode() more readable.
1371 -static void _dpaa2_eth_hw_add_uc_addr(const struct net_device *net_dev,
1372 - struct dpaa2_eth_priv *priv)
1373 +static void add_uc_hw_addr(const struct net_device *net_dev,
1374 + struct dpaa2_eth_priv *priv)
1376 struct netdev_hw_addr *ha;
1378 @@ -1257,17 +1284,18 @@ static void _dpaa2_eth_hw_add_uc_addr(co
1379 netdev_for_each_uc_addr(ha, net_dev) {
1380 err = dpni_add_mac_addr(priv->mc_io, 0, priv->mc_token,
1382 - DPAA2_ETH_WARN_IF_ERR(err, priv->net_dev,
1383 - "Could not add ucast MAC %pM to the filtering table (err %d)\n",
1386 + netdev_warn(priv->net_dev,
1387 + "Could not add ucast MAC %pM to the filtering table (err %d)\n",
1392 /* Copy mac multicast addresses from @net_dev to @priv
1393 * Its sole purpose is to make dpaa2_eth_set_rx_mode() more readable.
1395 -static void _dpaa2_eth_hw_add_mc_addr(const struct net_device *net_dev,
1396 - struct dpaa2_eth_priv *priv)
1397 +static void add_mc_hw_addr(const struct net_device *net_dev,
1398 + struct dpaa2_eth_priv *priv)
1400 struct netdev_hw_addr *ha;
1402 @@ -1275,9 +1303,10 @@ static void _dpaa2_eth_hw_add_mc_addr(co
1403 netdev_for_each_mc_addr(ha, net_dev) {
1404 err = dpni_add_mac_addr(priv->mc_io, 0, priv->mc_token,
1406 - DPAA2_ETH_WARN_IF_ERR(err, priv->net_dev,
1407 - "Could not add mcast MAC %pM to the filtering table (err %d)\n",
1410 + netdev_warn(priv->net_dev,
1411 + "Could not add mcast MAC %pM to the filtering table (err %d)\n",
1416 @@ -1296,11 +1325,11 @@ static void dpaa2_eth_set_rx_mode(struct
1417 /* Basic sanity checks; these probably indicate a misconfiguration */
1418 if (!(options & DPNI_OPT_UNICAST_FILTER) && max_uc != 0)
1419 netdev_info(net_dev,
1420 - "max_unicast_filters=%d, you must have DPNI_OPT_UNICAST_FILTER in the DPL\n",
1421 + "max_unicast_filters=%d, DPNI_OPT_UNICAST_FILTER option must be enabled\n",
1423 if (!(options & DPNI_OPT_MULTICAST_FILTER) && max_mc != 0)
1424 netdev_info(net_dev,
1425 - "max_multicast_filters=%d, you must have DPNI_OPT_MULTICAST_FILTER in the DPL\n",
1426 + "max_multicast_filters=%d, DPNI_OPT_MULTICAST_FILTER option must be enabled\n",
1429 /* Force promiscuous if the uc or mc counts exceed our capabilities. */
1430 @@ -1318,9 +1347,9 @@ static void dpaa2_eth_set_rx_mode(struct
1433 /* Adjust promisc settings due to flag combinations */
1434 - if (net_dev->flags & IFF_PROMISC) {
1435 + if (net_dev->flags & IFF_PROMISC)
1437 - } else if (net_dev->flags & IFF_ALLMULTI) {
1438 + if (net_dev->flags & IFF_ALLMULTI) {
1439 /* First, rebuild unicast filtering table. This should be done
1440 * in promisc mode, in order to avoid frame loss while we
1441 * progressively add entries to the table.
1442 @@ -1329,16 +1358,19 @@ static void dpaa2_eth_set_rx_mode(struct
1445 err = dpni_set_unicast_promisc(mc_io, 0, mc_token, 1);
1446 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't set uc promisc\n");
1448 + netdev_warn(net_dev, "Can't set uc promisc\n");
1450 /* Actual uc table reconstruction. */
1451 err = dpni_clear_mac_filters(mc_io, 0, mc_token, 1, 0);
1452 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't clear uc filters\n");
1453 - _dpaa2_eth_hw_add_uc_addr(net_dev, priv);
1455 + netdev_warn(net_dev, "Can't clear uc filters\n");
1456 + add_uc_hw_addr(net_dev, priv);
1458 /* Finally, clear uc promisc and set mc promisc as requested. */
1459 err = dpni_set_unicast_promisc(mc_io, 0, mc_token, 0);
1460 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't clear uc promisc\n");
1462 + netdev_warn(net_dev, "Can't clear uc promisc\n");
1463 goto force_mc_promisc;
1466 @@ -1346,32 +1378,39 @@ static void dpaa2_eth_set_rx_mode(struct
1467 * For now, rebuild mac filtering tables while forcing both of them on.
1469 err = dpni_set_unicast_promisc(mc_io, 0, mc_token, 1);
1470 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't set uc promisc (%d)\n", err);
1472 + netdev_warn(net_dev, "Can't set uc promisc (%d)\n", err);
1473 err = dpni_set_multicast_promisc(mc_io, 0, mc_token, 1);
1474 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't set mc promisc (%d)\n", err);
1476 + netdev_warn(net_dev, "Can't set mc promisc (%d)\n", err);
1478 /* Actual mac filtering tables reconstruction */
1479 err = dpni_clear_mac_filters(mc_io, 0, mc_token, 1, 1);
1480 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't clear mac filters\n");
1481 - _dpaa2_eth_hw_add_mc_addr(net_dev, priv);
1482 - _dpaa2_eth_hw_add_uc_addr(net_dev, priv);
1484 + netdev_warn(net_dev, "Can't clear mac filters\n");
1485 + add_mc_hw_addr(net_dev, priv);
1486 + add_uc_hw_addr(net_dev, priv);
1488 /* Now we can clear both ucast and mcast promisc, without risking
1489 * to drop legitimate frames anymore.
1491 err = dpni_set_unicast_promisc(mc_io, 0, mc_token, 0);
1492 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't clear ucast promisc\n");
1494 + netdev_warn(net_dev, "Can't clear ucast promisc\n");
1495 err = dpni_set_multicast_promisc(mc_io, 0, mc_token, 0);
1496 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't clear mcast promisc\n");
1498 + netdev_warn(net_dev, "Can't clear mcast promisc\n");
1503 err = dpni_set_unicast_promisc(mc_io, 0, mc_token, 1);
1504 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't set ucast promisc\n");
1506 + netdev_warn(net_dev, "Can't set ucast promisc\n");
1508 err = dpni_set_multicast_promisc(mc_io, 0, mc_token, 1);
1509 - DPAA2_ETH_WARN_IF_ERR(err, net_dev, "Can't set mcast promisc\n");
1511 + netdev_warn(net_dev, "Can't set mcast promisc\n");
1514 static int dpaa2_eth_set_features(struct net_device *net_dev,
1515 @@ -1379,20 +1418,19 @@ static int dpaa2_eth_set_features(struct
1517 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
1518 netdev_features_t changed = features ^ net_dev->features;
1522 if (changed & NETIF_F_RXCSUM) {
1523 - bool enable = !!(features & NETIF_F_RXCSUM);
1525 - err = dpaa2_eth_set_rx_csum(priv, enable);
1526 + enable = !!(features & NETIF_F_RXCSUM);
1527 + err = set_rx_csum(priv, enable);
1532 if (changed & (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM)) {
1533 - bool enable = !!(features &
1534 - (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM));
1535 - err = dpaa2_eth_set_tx_csum(priv, enable);
1536 + enable = !!(features & (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM));
1537 + err = set_tx_csum(priv, enable);
1541 @@ -1419,9 +1457,9 @@ static int dpaa2_eth_ts_ioctl(struct net
1545 - if (config.rx_filter == HWTSTAMP_FILTER_NONE)
1546 + if (config.rx_filter == HWTSTAMP_FILTER_NONE) {
1547 priv->ts_rx_en = false;
1550 priv->ts_rx_en = true;
1551 /* TS is set for all frame types, not only those requested */
1552 config.rx_filter = HWTSTAMP_FILTER_ALL;
1553 @@ -1435,8 +1473,8 @@ static int dpaa2_eth_ioctl(struct net_de
1555 if (cmd == SIOCSHWTSTAMP)
1556 return dpaa2_eth_ts_ioctl(dev, rq, cmd);
1563 static const struct net_device_ops dpaa2_eth_ops = {
1564 @@ -1452,7 +1490,7 @@ static const struct net_device_ops dpaa2
1565 .ndo_do_ioctl = dpaa2_eth_ioctl,
1568 -static void dpaa2_eth_cdan_cb(struct dpaa2_io_notification_ctx *ctx)
1569 +static void cdan_cb(struct dpaa2_io_notification_ctx *ctx)
1571 struct dpaa2_eth_channel *ch;
1573 @@ -1464,37 +1502,9 @@ static void dpaa2_eth_cdan_cb(struct dpa
1574 napi_schedule_irqoff(&ch->napi);
1577 -static void dpaa2_eth_setup_fqs(struct dpaa2_eth_priv *priv)
1581 - /* We have one TxConf FQ per Tx flow */
1582 - for (i = 0; i < priv->dpni_attrs.max_senders; i++) {
1583 - priv->fq[priv->num_fqs].netdev_priv = priv;
1584 - priv->fq[priv->num_fqs].type = DPAA2_TX_CONF_FQ;
1585 - priv->fq[priv->num_fqs].consume = dpaa2_eth_tx_conf;
1586 - priv->fq[priv->num_fqs++].flowid = DPNI_NEW_FLOW_ID;
1589 - /* The number of Rx queues (Rx distribution width) may be different from
1590 - * the number of cores.
1591 - * We only support one traffic class for now.
1593 - for (i = 0; i < dpaa2_queue_count(priv); i++) {
1594 - priv->fq[priv->num_fqs].netdev_priv = priv;
1595 - priv->fq[priv->num_fqs].type = DPAA2_RX_FQ;
1596 - priv->fq[priv->num_fqs].consume = dpaa2_eth_rx;
1597 - priv->fq[priv->num_fqs++].flowid = (u16)i;
1600 -#ifdef CONFIG_FSL_DPAA2_ETH_USE_ERR_QUEUE
1601 - /* We have exactly one Rx error queue per DPNI */
1602 - priv->fq[priv->num_fqs].netdev_priv = priv;
1603 - priv->fq[priv->num_fqs].type = DPAA2_RX_ERR_FQ;
1604 - priv->fq[priv->num_fqs++].consume = dpaa2_eth_rx_err;
1608 +/* Verify that the FLIB API version of various MC objects is supported
1611 static int check_obj_version(struct fsl_mc_device *ls_dev, u16 mc_version)
1613 char *name = ls_dev->obj_desc.type;
1614 @@ -1517,8 +1527,7 @@ static int check_obj_version(struct fsl_
1616 /* Check that the FLIB-defined version matches the one reported by MC */
1617 if (mc_version != flib_version) {
1619 - "%s FLIB version mismatch: MC reports %d, we have %d\n",
1620 + dev_err(dev, "%s FLIB version mismatch: MC reports %d, we have %d\n",
1621 name, mc_version, flib_version);
1624 @@ -1534,7 +1543,8 @@ static int check_obj_version(struct fsl_
1628 -static struct fsl_mc_device *dpaa2_dpcon_setup(struct dpaa2_eth_priv *priv)
1629 +/* Allocate and configure a DPCON object */
1630 +static struct fsl_mc_device *setup_dpcon(struct dpaa2_eth_priv *priv)
1632 struct fsl_mc_device *dpcon;
1633 struct device *dev = priv->net_dev->dev.parent;
1634 @@ -1582,8 +1592,8 @@ err_open:
1638 -static void dpaa2_dpcon_free(struct dpaa2_eth_priv *priv,
1639 - struct fsl_mc_device *dpcon)
1640 +static void free_dpcon(struct dpaa2_eth_priv *priv,
1641 + struct fsl_mc_device *dpcon)
1643 dpcon_disable(priv->mc_io, 0, dpcon->mc_handle);
1644 dpcon_close(priv->mc_io, 0, dpcon->mc_handle);
1645 @@ -1591,7 +1601,7 @@ static void dpaa2_dpcon_free(struct dpaa
1648 static struct dpaa2_eth_channel *
1649 -dpaa2_alloc_channel(struct dpaa2_eth_priv *priv)
1650 +alloc_channel(struct dpaa2_eth_priv *priv)
1652 struct dpaa2_eth_channel *channel;
1653 struct dpcon_attr attr;
1654 @@ -1599,12 +1609,10 @@ dpaa2_alloc_channel(struct dpaa2_eth_pri
1657 channel = kzalloc(sizeof(*channel), GFP_ATOMIC);
1659 - dev_err(dev, "Memory allocation failed\n");
1664 - channel->dpcon = dpaa2_dpcon_setup(priv);
1665 + channel->dpcon = setup_dpcon(priv);
1666 if (!channel->dpcon)
1669 @@ -1622,20 +1630,23 @@ dpaa2_alloc_channel(struct dpaa2_eth_pri
1673 - dpaa2_dpcon_free(priv, channel->dpcon);
1674 + free_dpcon(priv, channel->dpcon);
1680 -static void dpaa2_free_channel(struct dpaa2_eth_priv *priv,
1681 - struct dpaa2_eth_channel *channel)
1682 +static void free_channel(struct dpaa2_eth_priv *priv,
1683 + struct dpaa2_eth_channel *channel)
1685 - dpaa2_dpcon_free(priv, channel->dpcon);
1686 + free_dpcon(priv, channel->dpcon);
1690 -static int dpaa2_dpio_setup(struct dpaa2_eth_priv *priv)
1691 +/* DPIO setup: allocate and configure QBMan channels, setup core affinity
1692 + * and register data availability notifications
1694 +static int setup_dpio(struct dpaa2_eth_priv *priv)
1696 struct dpaa2_io_notification_ctx *nctx;
1697 struct dpaa2_eth_channel *channel;
1698 @@ -1652,7 +1663,7 @@ static int dpaa2_dpio_setup(struct dpaa2
1699 cpumask_clear(&priv->dpio_cpumask);
1700 for_each_online_cpu(i) {
1701 /* Try to allocate a channel */
1702 - channel = dpaa2_alloc_channel(priv);
1703 + channel = alloc_channel(priv);
1707 @@ -1660,7 +1671,7 @@ static int dpaa2_dpio_setup(struct dpaa2
1709 nctx = &channel->nctx;
1711 - nctx->cb = dpaa2_eth_cdan_cb;
1712 + nctx->cb = cdan_cb;
1713 nctx->id = channel->ch_id;
1714 nctx->desired_cpu = i;
1716 @@ -1671,7 +1682,7 @@ static int dpaa2_dpio_setup(struct dpaa2
1717 /* This core doesn't have an affine DPIO, but there's
1718 * a chance another one does, so keep trying
1720 - dpaa2_free_channel(priv, channel);
1721 + free_channel(priv, channel);
1725 @@ -1693,7 +1704,7 @@ static int dpaa2_dpio_setup(struct dpaa2
1726 cpumask_set_cpu(i, &priv->dpio_cpumask);
1727 priv->num_channels++;
1729 - if (priv->num_channels == dpaa2_max_channels(priv))
1730 + if (priv->num_channels == dpaa2_eth_max_channels(priv))
1734 @@ -1706,7 +1717,7 @@ static int dpaa2_dpio_setup(struct dpaa2
1737 dpaa2_io_service_deregister(NULL, nctx);
1738 - dpaa2_free_channel(priv, channel);
1739 + free_channel(priv, channel);
1741 if (cpumask_empty(&priv->dpio_cpumask)) {
1742 dev_err(dev, "No cpu with an affine DPIO/DPCON\n");
1743 @@ -1717,7 +1728,7 @@ err_alloc_ch:
1747 -static void dpaa2_dpio_free(struct dpaa2_eth_priv *priv)
1748 +static void free_dpio(struct dpaa2_eth_priv *priv)
1751 struct dpaa2_eth_channel *ch;
1752 @@ -1726,12 +1737,12 @@ static void dpaa2_dpio_free(struct dpaa2
1753 for (i = 0; i < priv->num_channels; i++) {
1754 ch = priv->channel[i];
1755 dpaa2_io_service_deregister(NULL, &ch->nctx);
1756 - dpaa2_free_channel(priv, ch);
1757 + free_channel(priv, ch);
1761 -static struct dpaa2_eth_channel *
1762 -dpaa2_get_channel_by_cpu(struct dpaa2_eth_priv *priv, int cpu)
1763 +static struct dpaa2_eth_channel *get_affine_channel(struct dpaa2_eth_priv *priv,
1766 struct device *dev = priv->net_dev->dev.parent;
1768 @@ -1748,11 +1759,11 @@ dpaa2_get_channel_by_cpu(struct dpaa2_et
1769 return priv->channel[0];
1772 -static void dpaa2_set_fq_affinity(struct dpaa2_eth_priv *priv)
1773 +static void set_fq_affinity(struct dpaa2_eth_priv *priv)
1775 struct device *dev = priv->net_dev->dev.parent;
1776 struct dpaa2_eth_fq *fq;
1777 - int rx_cpu, txconf_cpu;
1778 + int rx_cpu, txc_cpu;
1781 /* For each FQ, pick one channel/CPU to deliver frames to.
1782 @@ -1760,7 +1771,7 @@ static void dpaa2_set_fq_affinity(struct
1783 * through direct user intervention.
1785 rx_cpu = cpumask_first(&priv->dpio_cpumask);
1786 - txconf_cpu = cpumask_first(&priv->txconf_cpumask);
1787 + txc_cpu = cpumask_first(&priv->txconf_cpumask);
1789 for (i = 0; i < priv->num_fqs; i++) {
1791 @@ -1768,20 +1779,56 @@ static void dpaa2_set_fq_affinity(struct
1793 case DPAA2_RX_ERR_FQ:
1794 fq->target_cpu = rx_cpu;
1795 - cpumask_rr(rx_cpu, &priv->dpio_cpumask);
1796 + rx_cpu = cpumask_next(rx_cpu, &priv->dpio_cpumask);
1797 + if (rx_cpu >= nr_cpu_ids)
1798 + rx_cpu = cpumask_first(&priv->dpio_cpumask);
1800 case DPAA2_TX_CONF_FQ:
1801 - fq->target_cpu = txconf_cpu;
1802 - cpumask_rr(txconf_cpu, &priv->txconf_cpumask);
1803 + fq->target_cpu = txc_cpu;
1804 + txc_cpu = cpumask_next(txc_cpu, &priv->txconf_cpumask);
1805 + if (txc_cpu >= nr_cpu_ids)
1806 + txc_cpu = cpumask_first(&priv->txconf_cpumask);
1809 dev_err(dev, "Unknown FQ type: %d\n", fq->type);
1811 - fq->channel = dpaa2_get_channel_by_cpu(priv, fq->target_cpu);
1812 + fq->channel = get_affine_channel(priv, fq->target_cpu);
1816 +static void setup_fqs(struct dpaa2_eth_priv *priv)
1820 + /* We have one TxConf FQ per Tx flow */
1821 + for (i = 0; i < priv->dpni_attrs.max_senders; i++) {
1822 + priv->fq[priv->num_fqs].type = DPAA2_TX_CONF_FQ;
1823 + priv->fq[priv->num_fqs].consume = dpaa2_eth_tx_conf;
1824 + priv->fq[priv->num_fqs++].flowid = DPNI_NEW_FLOW_ID;
1827 + /* The number of Rx queues (Rx distribution width) may be different from
1828 + * the number of cores.
1829 + * We only support one traffic class for now.
1831 + for (i = 0; i < dpaa2_eth_queue_count(priv); i++) {
1832 + priv->fq[priv->num_fqs].type = DPAA2_RX_FQ;
1833 + priv->fq[priv->num_fqs].consume = dpaa2_eth_rx;
1834 + priv->fq[priv->num_fqs++].flowid = (u16)i;
1837 +#ifdef CONFIG_FSL_DPAA2_ETH_USE_ERR_QUEUE
1838 + /* We have exactly one Rx error queue per DPNI */
1839 + priv->fq[priv->num_fqs].type = DPAA2_RX_ERR_FQ;
1840 + priv->fq[priv->num_fqs++].consume = dpaa2_eth_rx_err;
1843 + /* For each FQ, decide on which core to process incoming frames */
1844 + set_fq_affinity(priv);
1847 -static int dpaa2_dpbp_setup(struct dpaa2_eth_priv *priv)
1848 +/* Allocate and configure one buffer pool for each interface */
1849 +static int setup_dpbp(struct dpaa2_eth_priv *priv)
1852 struct fsl_mc_device *dpbp_dev;
1853 @@ -1833,15 +1880,16 @@ err_open:
1857 -static void dpaa2_dpbp_free(struct dpaa2_eth_priv *priv)
1858 +static void free_dpbp(struct dpaa2_eth_priv *priv)
1860 - __dpaa2_dpbp_free(priv);
1862 dpbp_disable(priv->mc_io, 0, priv->dpbp_dev->mc_handle);
1863 dpbp_close(priv->mc_io, 0, priv->dpbp_dev->mc_handle);
1864 fsl_mc_object_free(priv->dpbp_dev);
1867 -static int dpaa2_dpni_setup(struct fsl_mc_device *ls_dev)
1868 +/* Configure the DPNI object this interface is associated with */
1869 +static int setup_dpni(struct fsl_mc_device *ls_dev)
1871 struct device *dev = &ls_dev->dev;
1872 struct dpaa2_eth_priv *priv;
1873 @@ -1854,7 +1902,7 @@ static int dpaa2_dpni_setup(struct fsl_m
1875 priv->dpni_id = ls_dev->obj_desc.id;
1877 - /* and get a handle for the DPNI this interface is associate with */
1878 + /* get a handle for the DPNI object */
1879 err = dpni_open(priv->mc_io, 0, priv->dpni_id, &priv->mc_token);
1881 dev_err(dev, "dpni_open() failed\n");
1882 @@ -1864,7 +1912,10 @@ static int dpaa2_dpni_setup(struct fsl_m
1883 ls_dev->mc_io = priv->mc_io;
1884 ls_dev->mc_handle = priv->mc_token;
1886 - dma_mem = kzalloc(DPAA2_EXT_CFG_SIZE, GFP_DMA | GFP_KERNEL);
1887 + /* Map a memory region which will be used by MC to pass us an
1888 + * attribute structure
1890 + dma_mem = kzalloc(DPAA2_EXT_CFG_SIZE, GFP_DMA | GFP_KERNEL);
1894 @@ -1878,10 +1929,15 @@ static int dpaa2_dpni_setup(struct fsl_m
1896 err = dpni_get_attributes(priv->mc_io, 0, priv->mc_token,
1899 + /* We'll check the return code after unmapping, as we need to
1902 + dma_unmap_single(dev, priv->dpni_attrs.ext_cfg_iova,
1903 + DPAA2_EXT_CFG_SIZE, DMA_FROM_DEVICE);
1906 dev_err(dev, "dpni_get_attributes() failed (err=%d)\n", err);
1907 - dma_unmap_single(dev, priv->dpni_attrs.ext_cfg_iova,
1908 - DPAA2_EXT_CFG_SIZE, DMA_FROM_DEVICE);
1912 @@ -1889,9 +1945,6 @@ static int dpaa2_dpni_setup(struct fsl_m
1916 - dma_unmap_single(dev, priv->dpni_attrs.ext_cfg_iova,
1917 - DPAA2_EXT_CFG_SIZE, DMA_FROM_DEVICE);
1919 memset(&priv->dpni_ext_cfg, 0, sizeof(priv->dpni_ext_cfg));
1920 err = dpni_extract_extended_cfg(&priv->dpni_ext_cfg, dma_mem);
1922 @@ -1909,15 +1962,15 @@ static int dpaa2_dpni_setup(struct fsl_m
1923 priv->buf_layout.private_data_size = DPAA2_ETH_SWA_SIZE;
1924 /* HW erratum mandates data alignment in multiples of 256 */
1925 priv->buf_layout.data_align = DPAA2_ETH_RX_BUF_ALIGN;
1929 err = dpni_set_rx_buffer_layout(priv->mc_io, 0, priv->mc_token,
1932 dev_err(dev, "dpni_set_rx_buffer_layout() failed");
1933 goto err_buf_layout;
1936 - /* remove Rx-only options */
1937 + /* tx buffer: remove Rx-only options */
1938 priv->buf_layout.options &= ~(DPNI_BUF_LAYOUT_OPT_DATA_ALIGN |
1939 DPNI_BUF_LAYOUT_OPT_PARSER_RESULT);
1940 err = dpni_set_tx_buffer_layout(priv->mc_io, 0, priv->mc_token,
1941 @@ -1926,7 +1979,7 @@ static int dpaa2_dpni_setup(struct fsl_m
1942 dev_err(dev, "dpni_set_tx_buffer_layout() failed");
1943 goto err_buf_layout;
1945 - /* ... tx-confirm. */
1946 + /* tx-confirm: same options as tx */
1947 priv->buf_layout.options &= ~DPNI_BUF_LAYOUT_OPT_PRIVATE_DATA_SIZE;
1948 priv->buf_layout.options |= DPNI_BUF_LAYOUT_OPT_TIMESTAMP;
1949 priv->buf_layout.pass_timestamp = 1;
1950 @@ -1946,8 +1999,9 @@ static int dpaa2_dpni_setup(struct fsl_m
1951 goto err_data_offset;
1954 - /* Warn in case TX data offset is not multiple of 64 bytes. */
1955 - WARN_ON(priv->tx_data_offset % 64);
1956 + if ((priv->tx_data_offset % 64) != 0)
1957 + dev_warn(dev, "Tx data offset (%d) not a multiple of 64B",
1958 + priv->tx_data_offset);
1960 /* Accommodate SWA space. */
1961 priv->tx_data_offset += DPAA2_ETH_SWA_SIZE;
1962 @@ -1976,7 +2030,7 @@ err_open:
1966 -static void dpaa2_dpni_free(struct dpaa2_eth_priv *priv)
1967 +static void free_dpni(struct dpaa2_eth_priv *priv)
1971 @@ -1988,8 +2042,8 @@ static void dpaa2_dpni_free(struct dpaa2
1972 dpni_close(priv->mc_io, 0, priv->mc_token);
1975 -static int dpaa2_rx_flow_setup(struct dpaa2_eth_priv *priv,
1976 - struct dpaa2_eth_fq *fq)
1977 +static int setup_rx_flow(struct dpaa2_eth_priv *priv,
1978 + struct dpaa2_eth_fq *fq)
1980 struct device *dev = priv->net_dev->dev.parent;
1981 struct dpni_queue_attr rx_queue_attr;
1982 @@ -2023,8 +2077,8 @@ static int dpaa2_rx_flow_setup(struct dp
1986 -static int dpaa2_tx_flow_setup(struct dpaa2_eth_priv *priv,
1987 - struct dpaa2_eth_fq *fq)
1988 +static int setup_tx_flow(struct dpaa2_eth_priv *priv,
1989 + struct dpaa2_eth_fq *fq)
1991 struct device *dev = priv->net_dev->dev.parent;
1992 struct dpni_tx_flow_cfg tx_flow_cfg;
1993 @@ -2070,15 +2124,16 @@ static int dpaa2_tx_flow_setup(struct dp
1996 #ifdef CONFIG_FSL_DPAA2_ETH_USE_ERR_QUEUE
1997 -static int dpaa2_rx_err_setup(struct dpaa2_eth_priv *priv,
1998 - struct dpaa2_eth_fq *fq)
1999 +static int setup_rx_err_flow(struct dpaa2_eth_priv *priv,
2000 + struct dpaa2_eth_fq *fq)
2002 struct dpni_queue_attr queue_attr;
2003 struct dpni_queue_cfg queue_cfg;
2006 /* Configure the Rx error queue to generate CDANs,
2007 - * just like the Rx queues */
2008 + * just like the Rx queues
2010 queue_cfg.options = DPNI_QUEUE_OPT_USER_CTX | DPNI_QUEUE_OPT_DEST;
2011 queue_cfg.dest_cfg.dest_type = DPNI_DEST_DPCON;
2012 queue_cfg.dest_cfg.priority = 1;
2013 @@ -2091,7 +2146,8 @@ static int dpaa2_rx_err_setup(struct dpa
2017 - err = dpni_get_rx_err_queue(priv->mc_io, 0, priv->mc_token, &queue_attr);
2018 + err = dpni_get_rx_err_queue(priv->mc_io, 0, priv->mc_token,
2021 netdev_err(priv->net_dev, "dpni_get_rx_err_queue() failed\n");
2023 @@ -2102,7 +2158,10 @@ static int dpaa2_rx_err_setup(struct dpa
2027 -static int dpaa2_dpni_bind(struct dpaa2_eth_priv *priv)
2028 +/* Bind the DPNI to its needed objects and resources: buffer pool, DPIOs,
2029 + * frame queues and channels
2031 +static int bind_dpni(struct dpaa2_eth_priv *priv)
2033 struct net_device *net_dev = priv->net_dev;
2034 struct device *dev = net_dev->dev.parent;
2035 @@ -2114,20 +2173,20 @@ static int dpaa2_dpni_bind(struct dpaa2_
2036 pools_params.num_dpbp = 1;
2037 pools_params.pools[0].dpbp_id = priv->dpbp_dev->obj_desc.id;
2038 pools_params.pools[0].backup_pool = 0;
2039 - pools_params.pools[0].buffer_size = DPAA2_ETH_RX_BUFFER_SIZE;
2040 + pools_params.pools[0].buffer_size = DPAA2_ETH_RX_BUF_SIZE;
2041 err = dpni_set_pools(priv->mc_io, 0, priv->mc_token, &pools_params);
2043 dev_err(dev, "dpni_set_pools() failed\n");
2047 - dpaa2_cls_check(net_dev);
2048 + check_fs_support(net_dev);
2050 /* have the interface implicitly distribute traffic based on supported
2053 if (dpaa2_eth_hash_enabled(priv)) {
2054 - err = dpaa2_set_hash(net_dev, DPAA2_RXH_SUPPORTED);
2055 + err = dpaa2_eth_set_hash(net_dev, DPAA2_RXH_SUPPORTED);
2059 @@ -2151,14 +2210,14 @@ static int dpaa2_dpni_bind(struct dpaa2_
2060 for (i = 0; i < priv->num_fqs; i++) {
2061 switch (priv->fq[i].type) {
2063 - err = dpaa2_rx_flow_setup(priv, &priv->fq[i]);
2064 + err = setup_rx_flow(priv, &priv->fq[i]);
2066 case DPAA2_TX_CONF_FQ:
2067 - err = dpaa2_tx_flow_setup(priv, &priv->fq[i]);
2068 + err = setup_tx_flow(priv, &priv->fq[i]);
2070 #ifdef CONFIG_FSL_DPAA2_ETH_USE_ERR_QUEUE
2071 case DPAA2_RX_ERR_FQ:
2072 - err = dpaa2_rx_err_setup(priv, &priv->fq[i]);
2073 + err = setup_rx_err_flow(priv, &priv->fq[i]);
2077 @@ -2178,7 +2237,8 @@ static int dpaa2_dpni_bind(struct dpaa2_
2081 -static int dpaa2_eth_alloc_rings(struct dpaa2_eth_priv *priv)
2082 +/* Allocate rings for storing incoming frame descriptors */
2083 +static int alloc_rings(struct dpaa2_eth_priv *priv)
2085 struct net_device *net_dev = priv->net_dev;
2086 struct device *dev = net_dev->dev.parent;
2087 @@ -2205,7 +2265,7 @@ err_ring:
2091 -static void dpaa2_eth_free_rings(struct dpaa2_eth_priv *priv)
2092 +static void free_rings(struct dpaa2_eth_priv *priv)
2096 @@ -2213,7 +2273,7 @@ static void dpaa2_eth_free_rings(struct
2097 dpaa2_io_store_destroy(priv->channel[i]->store);
2100 -static int dpaa2_eth_netdev_init(struct net_device *net_dev)
2101 +static int netdev_init(struct net_device *net_dev)
2104 struct device *dev = net_dev->dev.parent;
2105 @@ -2223,7 +2283,9 @@ static int dpaa2_eth_netdev_init(struct
2107 net_dev->netdev_ops = &dpaa2_eth_ops;
2109 - /* If the DPL contains all-0 mac_addr, set a random hardware address */
2110 + /* If the DPNI attributes contain an all-0 mac_addr,
2111 + * set a random hardware address
2113 err = dpni_get_primary_mac_addr(priv->mc_io, 0, priv->mc_token,
2116 @@ -2281,14 +2343,13 @@ static int dpaa2_eth_netdev_init(struct
2120 -#ifdef CONFIG_FSL_DPAA2_ETH_LINK_POLL
2121 -static int dpaa2_poll_link_state(void *arg)
2122 +static int poll_link_state(void *arg)
2124 struct dpaa2_eth_priv *priv = (struct dpaa2_eth_priv *)arg;
2127 while (!kthread_should_stop()) {
2128 - err = dpaa2_link_state_update(priv);
2129 + err = link_state_update(priv);
2133 @@ -2297,7 +2358,7 @@ static int dpaa2_poll_link_state(void *a
2139 static irqreturn_t dpni_irq0_handler(int irq_num, void *arg)
2141 return IRQ_WAKE_THREAD;
2142 @@ -2312,7 +2373,6 @@ static irqreturn_t dpni_irq0_handler_thr
2143 struct net_device *net_dev = dev_get_drvdata(dev);
2146 - netdev_dbg(net_dev, "IRQ %d received\n", irq_num);
2147 err = dpni_get_irq_status(dpni_dev->mc_io, 0, dpni_dev->mc_handle,
2148 irq_index, &status);
2149 if (unlikely(err)) {
2150 @@ -2323,7 +2383,7 @@ static irqreturn_t dpni_irq0_handler_thr
2152 if (status & DPNI_IRQ_EVENT_LINK_CHANGED) {
2153 clear |= DPNI_IRQ_EVENT_LINK_CHANGED;
2154 - dpaa2_link_state_update(netdev_priv(net_dev));
2155 + link_state_update(netdev_priv(net_dev));
2159 @@ -2332,17 +2392,18 @@ out:
2163 -static int dpaa2_eth_setup_irqs(struct fsl_mc_device *ls_dev)
2164 +static int setup_irqs(struct fsl_mc_device *ls_dev)
2167 struct fsl_mc_device_irq *irq;
2168 - int irq_count = ls_dev->obj_desc.irq_count;
2169 u8 irq_index = DPNI_IRQ_INDEX;
2170 u32 mask = DPNI_IRQ_EVENT_LINK_CHANGED;
2172 - /* The only interrupt supported now is the link state notification. */
2173 - if (WARN_ON(irq_count != 1))
2175 + err = fsl_mc_allocate_irqs(ls_dev);
2177 + dev_err(&ls_dev->dev, "MC irqs allocation failed\n");
2181 irq = ls_dev->irqs[0];
2182 err = devm_request_threaded_irq(&ls_dev->dev, irq->msi_desc->irq,
2183 @@ -2352,28 +2413,34 @@ static int dpaa2_eth_setup_irqs(struct f
2184 dev_name(&ls_dev->dev), &ls_dev->dev);
2186 dev_err(&ls_dev->dev, "devm_request_threaded_irq(): %d", err);
2191 err = dpni_set_irq_mask(ls_dev->mc_io, 0, ls_dev->mc_handle,
2194 dev_err(&ls_dev->dev, "dpni_set_irq_mask(): %d", err);
2199 err = dpni_set_irq_enable(ls_dev->mc_io, 0, ls_dev->mc_handle,
2202 dev_err(&ls_dev->dev, "dpni_set_irq_enable(): %d", err);
2210 + devm_free_irq(&ls_dev->dev, irq->msi_desc->irq, &ls_dev->dev);
2212 + fsl_mc_free_irqs(ls_dev);
2218 -static void dpaa2_eth_napi_add(struct dpaa2_eth_priv *priv)
2219 +static void add_ch_napi(struct dpaa2_eth_priv *priv)
2222 struct dpaa2_eth_channel *ch;
2223 @@ -2386,7 +2453,7 @@ static void dpaa2_eth_napi_add(struct dp
2227 -static void dpaa2_eth_napi_del(struct dpaa2_eth_priv *priv)
2228 +static void del_ch_napi(struct dpaa2_eth_priv *priv)
2231 struct dpaa2_eth_channel *ch;
2232 @@ -2398,7 +2465,6 @@ static void dpaa2_eth_napi_del(struct dp
2237 static ssize_t dpaa2_eth_show_tx_shaping(struct device *dev,
2238 struct device_attribute *attr,
2240 @@ -2482,22 +2548,21 @@ static ssize_t dpaa2_eth_write_txconf_cp
2243 /* Set the new TxConf FQ affinities */
2244 - dpaa2_set_fq_affinity(priv);
2245 + set_fq_affinity(priv);
2247 -#ifdef CONFIG_FSL_DPAA2_ETH_LINK_POLL
2248 /* dpaa2_eth_open() below will *stop* the Tx queues until an explicit
2249 * link up notification is received. Give the polling thread enough time
2250 * to detect the link state change, or else we'll end up with the
2251 * transmission side forever shut down.
2253 - msleep(2 * DPAA2_ETH_LINK_STATE_REFRESH);
2255 + if (priv->do_link_poll)
2256 + msleep(2 * DPAA2_ETH_LINK_STATE_REFRESH);
2258 for (i = 0; i < priv->num_fqs; i++) {
2260 if (fq->type != DPAA2_TX_CONF_FQ)
2262 - dpaa2_tx_flow_setup(priv, fq);
2263 + setup_tx_flow(priv, fq);
2267 @@ -2568,7 +2633,6 @@ static int dpaa2_eth_probe(struct fsl_mc
2269 priv = netdev_priv(net_dev);
2270 priv->net_dev = net_dev;
2271 - priv->msg_enable = netif_msg_init(debug, -1);
2273 /* Obtain a MC portal */
2274 err = fsl_mc_portal_allocate(dpni_dev, FSL_MC_IO_ATOMIC_CONTEXT_PORTAL,
2275 @@ -2578,39 +2642,27 @@ static int dpaa2_eth_probe(struct fsl_mc
2276 goto err_portal_alloc;
2279 -#ifndef CONFIG_FSL_DPAA2_ETH_LINK_POLL
2280 - err = fsl_mc_allocate_irqs(dpni_dev);
2282 - dev_err(dev, "MC irqs allocation failed\n");
2283 - goto err_irqs_alloc;
2287 - /* DPNI initialization */
2288 - err = dpaa2_dpni_setup(dpni_dev);
2290 + /* MC objects initialization and configuration */
2291 + err = setup_dpni(dpni_dev);
2293 goto err_dpni_setup;
2296 - err = dpaa2_dpio_setup(priv);
2297 + err = setup_dpio(priv);
2299 goto err_dpio_setup;
2302 - dpaa2_eth_setup_fqs(priv);
2303 - dpaa2_set_fq_affinity(priv);
2307 - err = dpaa2_dpbp_setup(priv);
2308 + err = setup_dpbp(priv);
2310 goto err_dpbp_setup;
2312 - /* DPNI binding to DPIO and DPBPs */
2313 - err = dpaa2_dpni_bind(priv);
2314 + err = bind_dpni(priv);
2318 - dpaa2_eth_napi_add(priv);
2319 + /* Add a NAPI context for each channel */
2320 + add_ch_napi(priv);
2322 /* Percpu statistics */
2323 priv->percpu_stats = alloc_percpu(*priv->percpu_stats);
2324 @@ -2635,38 +2687,37 @@ static int dpaa2_eth_probe(struct fsl_mc
2325 dev_warn(&net_dev->dev, "using name \"%s\"\n", net_dev->name);
2328 - err = dpaa2_eth_netdev_init(net_dev);
2329 + err = netdev_init(net_dev);
2331 goto err_netdev_init;
2333 /* Configure checksum offload based on current interface flags */
2334 - err = dpaa2_eth_set_rx_csum(priv,
2335 - !!(net_dev->features & NETIF_F_RXCSUM));
2336 + err = set_rx_csum(priv, !!(net_dev->features & NETIF_F_RXCSUM));
2340 - err = dpaa2_eth_set_tx_csum(priv,
2341 - !!(net_dev->features &
2342 - (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM)));
2343 + err = set_tx_csum(priv, !!(net_dev->features &
2344 + (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM)));
2348 - err = dpaa2_eth_alloc_rings(priv);
2349 + err = alloc_rings(priv);
2351 goto err_alloc_rings;
2353 net_dev->ethtool_ops = &dpaa2_ethtool_ops;
2355 -#ifdef CONFIG_FSL_DPAA2_ETH_LINK_POLL
2356 - priv->poll_thread = kthread_run(dpaa2_poll_link_state, priv,
2357 - "%s_poll_link", net_dev->name);
2359 - err = dpaa2_eth_setup_irqs(dpni_dev);
2360 + err = setup_irqs(dpni_dev);
2362 - netdev_err(net_dev, "ERROR %d setting up interrupts", err);
2363 - goto err_setup_irqs;
2364 + netdev_warn(net_dev, "Failed to set link interrupt, fall back to polling\n");
2365 + priv->poll_thread = kthread_run(poll_link_state, priv,
2366 + "%s_poll_link", net_dev->name);
2367 + if (IS_ERR(priv->poll_thread)) {
2368 + netdev_err(net_dev, "Error starting polling thread\n");
2369 + goto err_poll_thread;
2371 + priv->do_link_poll = true;
2375 dpaa2_eth_sysfs_init(&net_dev->dev);
2376 dpaa2_dbg_add(priv);
2377 @@ -2674,10 +2725,8 @@ static int dpaa2_eth_probe(struct fsl_mc
2378 dev_info(dev, "Probed interface %s\n", net_dev->name);
2381 -#ifndef CONFIG_FSL_DPAA2_ETH_LINK_POLL
2384 - dpaa2_eth_free_rings(priv);
2389 unregister_netdev(net_dev);
2390 @@ -2686,19 +2735,15 @@ err_netdev_init:
2391 err_alloc_percpu_extras:
2392 free_percpu(priv->percpu_stats);
2393 err_alloc_percpu_stats:
2394 - dpaa2_eth_napi_del(priv);
2395 + del_ch_napi(priv);
2397 - dpaa2_dpbp_free(priv);
2400 - dpaa2_dpio_free(priv);
2403 kfree(priv->cls_rule);
2404 dpni_close(priv->mc_io, 0, priv->mc_token);
2406 -#ifndef CONFIG_FSL_DPAA2_ETH_LINK_POLL
2407 - fsl_mc_free_irqs(dpni_dev);
2410 fsl_mc_portal_free(priv->mc_io);
2412 dev_set_drvdata(dev, NULL);
2413 @@ -2723,22 +2768,21 @@ static int dpaa2_eth_remove(struct fsl_m
2414 unregister_netdev(net_dev);
2415 dev_info(net_dev->dev.parent, "Removed interface %s\n", net_dev->name);
2417 - dpaa2_dpio_free(priv);
2418 - dpaa2_eth_free_rings(priv);
2419 - dpaa2_eth_napi_del(priv);
2420 - dpaa2_dpbp_free(priv);
2421 - dpaa2_dpni_free(priv);
2424 + del_ch_napi(priv);
2428 fsl_mc_portal_free(priv->mc_io);
2430 free_percpu(priv->percpu_stats);
2431 free_percpu(priv->percpu_extras);
2433 -#ifdef CONFIG_FSL_DPAA2_ETH_LINK_POLL
2434 - kthread_stop(priv->poll_thread);
2436 - fsl_mc_free_irqs(ls_dev);
2438 + if (priv->do_link_poll)
2439 + kthread_stop(priv->poll_thread);
2441 + fsl_mc_free_irqs(ls_dev);
2443 kfree(priv->cls_rule);
2445 --- a/drivers/staging/fsl-dpaa2/ethernet/dpaa2-eth.h
2446 +++ b/drivers/staging/fsl-dpaa2/ethernet/dpaa2-eth.h
2449 #define DPAA2_ETH_STORE_SIZE 16
2451 -/* Maximum receive frame size is 64K */
2452 -#define DPAA2_ETH_MAX_SG_ENTRIES ((64 * 1024) / DPAA2_ETH_RX_BUFFER_SIZE)
2453 +/* Maximum number of scatter-gather entries in an ingress frame,
2454 + * considering the maximum receive frame size is 64K
2456 +#define DPAA2_ETH_MAX_SG_ENTRIES ((64 * 1024) / DPAA2_ETH_RX_BUF_SIZE)
2458 /* Maximum acceptable MTU value. It is in direct relation with the MC-enforced
2459 * Max Frame Length (currently 10k).
2461 #define DPAA2_ETH_NUM_BUFS (DPAA2_ETH_MAX_FRAMES_PER_QUEUE + 256)
2462 #define DPAA2_ETH_REFILL_THRESH DPAA2_ETH_MAX_FRAMES_PER_QUEUE
2464 +/* Maximum number of buffers that can be acquired/released through a single
2467 +#define DPAA2_ETH_BUFS_PER_CMD 7
2469 /* Hardware requires alignment for ingress/egress buffer addresses
2470 * and ingress buffer lengths.
2472 -#define DPAA2_ETH_RX_BUFFER_SIZE 2048
2473 +#define DPAA2_ETH_RX_BUF_SIZE 2048
2474 #define DPAA2_ETH_TX_BUF_ALIGN 64
2475 #define DPAA2_ETH_RX_BUF_ALIGN 256
2476 #define DPAA2_ETH_NEEDED_HEADROOM(p_priv) \
2477 ((p_priv)->tx_data_offset + DPAA2_ETH_TX_BUF_ALIGN)
2479 +/* Hardware only sees DPAA2_ETH_RX_BUF_SIZE, but we need to allocate ingress
2480 + * buffers large enough to allow building an skb around them and also account
2481 + * for alignment restrictions
2483 #define DPAA2_ETH_BUF_RAW_SIZE \
2484 - (DPAA2_ETH_RX_BUFFER_SIZE + \
2485 + (DPAA2_ETH_RX_BUF_SIZE + \
2486 SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) + \
2487 DPAA2_ETH_RX_BUF_ALIGN)
2489 @@ -127,57 +138,56 @@ struct dpaa2_fas {
2493 +/* Error and status bits in the frame annotation status word */
2494 /* Debug frame, otherwise supposed to be discarded */
2495 -#define DPAA2_ETH_FAS_DISC 0x80000000
2496 +#define DPAA2_FAS_DISC 0x80000000
2498 -#define DPAA2_ETH_FAS_MS 0x40000000
2499 -#define DPAA2_ETH_FAS_PTP 0x08000000
2500 +#define DPAA2_FAS_MS 0x40000000
2501 +#define DPAA2_FAS_PTP 0x08000000
2502 /* Ethernet multicast frame */
2503 -#define DPAA2_ETH_FAS_MC 0x04000000
2504 +#define DPAA2_FAS_MC 0x04000000
2505 /* Ethernet broadcast frame */
2506 -#define DPAA2_ETH_FAS_BC 0x02000000
2507 -#define DPAA2_ETH_FAS_KSE 0x00040000
2508 -#define DPAA2_ETH_FAS_EOFHE 0x00020000
2509 -#define DPAA2_ETH_FAS_MNLE 0x00010000
2510 -#define DPAA2_ETH_FAS_TIDE 0x00008000
2511 -#define DPAA2_ETH_FAS_PIEE 0x00004000
2512 +#define DPAA2_FAS_BC 0x02000000
2513 +#define DPAA2_FAS_KSE 0x00040000
2514 +#define DPAA2_FAS_EOFHE 0x00020000
2515 +#define DPAA2_FAS_MNLE 0x00010000
2516 +#define DPAA2_FAS_TIDE 0x00008000
2517 +#define DPAA2_FAS_PIEE 0x00004000
2518 /* Frame length error */
2519 -#define DPAA2_ETH_FAS_FLE 0x00002000
2520 -/* Frame physical error; our favourite pastime */
2521 -#define DPAA2_ETH_FAS_FPE 0x00001000
2522 -#define DPAA2_ETH_FAS_PTE 0x00000080
2523 -#define DPAA2_ETH_FAS_ISP 0x00000040
2524 -#define DPAA2_ETH_FAS_PHE 0x00000020
2525 -#define DPAA2_ETH_FAS_BLE 0x00000010
2526 +#define DPAA2_FAS_FLE 0x00002000
2527 +/* Frame physical error */
2528 +#define DPAA2_FAS_FPE 0x00001000
2529 +#define DPAA2_FAS_PTE 0x00000080
2530 +#define DPAA2_FAS_ISP 0x00000040
2531 +#define DPAA2_FAS_PHE 0x00000020
2532 +#define DPAA2_FAS_BLE 0x00000010
2533 /* L3 csum validation performed */
2534 -#define DPAA2_ETH_FAS_L3CV 0x00000008
2535 +#define DPAA2_FAS_L3CV 0x00000008
2537 -#define DPAA2_ETH_FAS_L3CE 0x00000004
2538 +#define DPAA2_FAS_L3CE 0x00000004
2539 /* L4 csum validation performed */
2540 -#define DPAA2_ETH_FAS_L4CV 0x00000002
2541 +#define DPAA2_FAS_L4CV 0x00000002
2543 -#define DPAA2_ETH_FAS_L4CE 0x00000001
2544 -/* These bits always signal errors */
2545 -#define DPAA2_ETH_RX_ERR_MASK (DPAA2_ETH_FAS_KSE | \
2546 - DPAA2_ETH_FAS_EOFHE | \
2547 - DPAA2_ETH_FAS_MNLE | \
2548 - DPAA2_ETH_FAS_TIDE | \
2549 - DPAA2_ETH_FAS_PIEE | \
2550 - DPAA2_ETH_FAS_FLE | \
2551 - DPAA2_ETH_FAS_FPE | \
2552 - DPAA2_ETH_FAS_PTE | \
2553 - DPAA2_ETH_FAS_ISP | \
2554 - DPAA2_ETH_FAS_PHE | \
2555 - DPAA2_ETH_FAS_BLE | \
2556 - DPAA2_ETH_FAS_L3CE | \
2557 - DPAA2_ETH_FAS_L4CE)
2558 -/* Unsupported features in the ingress */
2559 -#define DPAA2_ETH_RX_UNSUPP_MASK DPAA2_ETH_FAS_MS
2560 +#define DPAA2_FAS_L4CE 0x00000001
2561 +/* Possible errors on the ingress path */
2562 +#define DPAA2_ETH_RX_ERR_MASK (DPAA2_FAS_KSE | \
2563 + DPAA2_FAS_EOFHE | \
2564 + DPAA2_FAS_MNLE | \
2565 + DPAA2_FAS_TIDE | \
2566 + DPAA2_FAS_PIEE | \
2573 + DPAA2_FAS_L3CE | \
2576 -#define DPAA2_ETH_TXCONF_ERR_MASK (DPAA2_ETH_FAS_KSE | \
2577 - DPAA2_ETH_FAS_EOFHE | \
2578 - DPAA2_ETH_FAS_MNLE | \
2579 - DPAA2_ETH_FAS_TIDE)
2580 +#define DPAA2_ETH_TXCONF_ERR_MASK (DPAA2_FAS_KSE | \
2581 + DPAA2_FAS_EOFHE | \
2582 + DPAA2_FAS_MNLE | \
2585 /* Time in milliseconds between link state updates */
2586 #define DPAA2_ETH_LINK_STATE_REFRESH 1000
2587 @@ -185,7 +195,7 @@ struct dpaa2_fas {
2588 /* Driver statistics, other than those in struct rtnl_link_stats64.
2589 * These are usually collected per-CPU and aggregated by ethtool.
2591 -struct dpaa2_eth_stats {
2592 +struct dpaa2_eth_drv_stats {
2593 __u64 tx_conf_frames;
2594 __u64 tx_conf_bytes;
2596 @@ -210,15 +220,17 @@ struct dpaa2_eth_ch_stats {
2598 /* Number of frames received on queues from this channel */
2604 -/* Maximum number of Rx queues associated with a DPNI */
2605 +/* Maximum number of queues associated with a DPNI */
2606 #define DPAA2_ETH_MAX_RX_QUEUES 16
2607 #define DPAA2_ETH_MAX_TX_QUEUES NR_CPUS
2608 #define DPAA2_ETH_MAX_RX_ERR_QUEUES 1
2609 -#define DPAA2_ETH_MAX_QUEUES (DPAA2_ETH_MAX_RX_QUEUES + \
2610 - DPAA2_ETH_MAX_TX_QUEUES + \
2611 - DPAA2_ETH_MAX_RX_ERR_QUEUES)
2612 +#define DPAA2_ETH_MAX_QUEUES (DPAA2_ETH_MAX_RX_QUEUES + \
2613 + DPAA2_ETH_MAX_TX_QUEUES + \
2614 + DPAA2_ETH_MAX_RX_ERR_QUEUES)
2616 #define DPAA2_ETH_MAX_DPCONS NR_CPUS
2618 @@ -241,7 +253,6 @@ struct dpaa2_eth_fq {
2619 struct dpaa2_eth_channel *,
2620 const struct dpaa2_fd *,
2621 struct napi_struct *);
2622 - struct dpaa2_eth_priv *netdev_priv; /* backpointer */
2623 struct dpaa2_eth_fq_stats stats;
2626 @@ -258,16 +269,16 @@ struct dpaa2_eth_channel {
2627 struct dpaa2_eth_ch_stats stats;
2630 -struct dpaa2_cls_rule {
2631 +struct dpaa2_eth_cls_rule {
2632 struct ethtool_rx_flow_spec fs;
2636 +/* Driver private data */
2637 struct dpaa2_eth_priv {
2638 struct net_device *net_dev;
2641 - /* First queue is tx conf, the rest are rx */
2642 struct dpaa2_eth_fq fq[DPAA2_ETH_MAX_QUEUES];
2645 @@ -299,12 +310,12 @@ struct dpaa2_eth_priv {
2646 /* Standard statistics */
2647 struct rtnl_link_stats64 __percpu *percpu_stats;
2648 /* Extra stats, in addition to the ones known by the kernel */
2649 - struct dpaa2_eth_stats __percpu *percpu_extras;
2650 - u32 msg_enable; /* net_device message level */
2651 + struct dpaa2_eth_drv_stats __percpu *percpu_extras;
2655 struct dpni_link_state link_state;
2656 + bool do_link_poll;
2657 struct task_struct *poll_thread;
2659 /* enabled ethtool hashing bits */
2660 @@ -315,7 +326,7 @@ struct dpaa2_eth_priv {
2663 /* array of classification rules */
2664 - struct dpaa2_cls_rule *cls_rule;
2665 + struct dpaa2_eth_cls_rule *cls_rule;
2667 struct dpni_tx_shaping_cfg shaping_cfg;
2669 @@ -341,9 +352,9 @@ struct dpaa2_eth_priv {
2671 extern const struct ethtool_ops dpaa2_ethtool_ops;
2673 -int dpaa2_set_hash(struct net_device *net_dev, u64 flags);
2674 +int dpaa2_eth_set_hash(struct net_device *net_dev, u64 flags);
2676 -static int dpaa2_queue_count(struct dpaa2_eth_priv *priv)
2677 +static int dpaa2_eth_queue_count(struct dpaa2_eth_priv *priv)
2679 if (!dpaa2_eth_hash_enabled(priv))
2681 @@ -351,16 +362,16 @@ static int dpaa2_queue_count(struct dpaa
2682 return priv->dpni_ext_cfg.tc_cfg[0].max_dist;
2685 -static inline int dpaa2_max_channels(struct dpaa2_eth_priv *priv)
2686 +static inline int dpaa2_eth_max_channels(struct dpaa2_eth_priv *priv)
2688 /* Ideally, we want a number of channels large enough
2689 * to accommodate both the Rx distribution size
2690 * and the max number of Tx confirmation queues
2692 - return max_t(int, dpaa2_queue_count(priv),
2693 + return max_t(int, dpaa2_eth_queue_count(priv),
2694 priv->dpni_attrs.max_senders);
2697 -void dpaa2_cls_check(struct net_device *);
2698 +void check_fs_support(struct net_device *);
2700 #endif /* __DPAA2_H */
2701 --- a/drivers/staging/fsl-dpaa2/ethernet/dpaa2-ethtool.c
2702 +++ b/drivers/staging/fsl-dpaa2/ethernet/dpaa2-ethtool.c
2703 @@ -52,7 +52,7 @@ char dpaa2_ethtool_stats[][ETH_GSTRING_L
2705 #define DPAA2_ETH_NUM_STATS ARRAY_SIZE(dpaa2_ethtool_stats)
2707 -/* To be kept in sync with 'struct dpaa2_eth_stats' */
2708 +/* To be kept in sync with 'struct dpaa2_eth_drv_stats' */
2709 char dpaa2_ethtool_extras[][ETH_GSTRING_LEN] = {
2712 @@ -63,12 +63,12 @@ char dpaa2_ethtool_extras[][ETH_GSTRING_
2715 /* how many times we had to retry the enqueue command */
2717 + "enqueue portal busy",
2721 /* How many times we had to retry the volatile dequeue command */
2723 + "dequeue portal busy",
2724 + "channel pull errors",
2725 /* Number of notifications received */
2727 #ifdef CONFIG_FSL_QBMAN_DEBUG
2728 @@ -83,8 +83,8 @@ char dpaa2_ethtool_extras[][ETH_GSTRING_
2730 #define DPAA2_ETH_NUM_EXTRA_STATS ARRAY_SIZE(dpaa2_ethtool_extras)
2732 -static void dpaa2_get_drvinfo(struct net_device *net_dev,
2733 - struct ethtool_drvinfo *drvinfo)
2734 +static void dpaa2_eth_get_drvinfo(struct net_device *net_dev,
2735 + struct ethtool_drvinfo *drvinfo)
2737 struct mc_version mc_ver;
2738 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
2739 @@ -112,20 +112,8 @@ static void dpaa2_get_drvinfo(struct net
2740 sizeof(drvinfo->bus_info));
2743 -static u32 dpaa2_get_msglevel(struct net_device *net_dev)
2745 - return ((struct dpaa2_eth_priv *)netdev_priv(net_dev))->msg_enable;
2748 -static void dpaa2_set_msglevel(struct net_device *net_dev,
2751 - ((struct dpaa2_eth_priv *)netdev_priv(net_dev))->msg_enable =
2755 -static int dpaa2_get_settings(struct net_device *net_dev,
2756 - struct ethtool_cmd *cmd)
2757 +static int dpaa2_eth_get_settings(struct net_device *net_dev,
2758 + struct ethtool_cmd *cmd)
2760 struct dpni_link_state state = {0};
2762 @@ -152,8 +140,8 @@ out:
2766 -static int dpaa2_set_settings(struct net_device *net_dev,
2767 - struct ethtool_cmd *cmd)
2768 +static int dpaa2_eth_set_settings(struct net_device *net_dev,
2769 + struct ethtool_cmd *cmd)
2771 struct dpni_link_cfg cfg = {0};
2772 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
2773 @@ -190,8 +178,8 @@ static int dpaa2_set_settings(struct net
2777 -static void dpaa2_get_strings(struct net_device *netdev, u32 stringset,
2779 +static void dpaa2_eth_get_strings(struct net_device *netdev, u32 stringset,
2784 @@ -210,7 +198,7 @@ static void dpaa2_get_strings(struct net
2788 -static int dpaa2_get_sset_count(struct net_device *net_dev, int sset)
2789 +static int dpaa2_eth_get_sset_count(struct net_device *net_dev, int sset)
2792 case ETH_SS_STATS: /* ethtool_get_stats(), ethtool_get_drvinfo() */
2793 @@ -222,9 +210,9 @@ static int dpaa2_get_sset_count(struct n
2795 /** Fill in hardware counters, as returned by the MC firmware.
2797 -static void dpaa2_get_ethtool_stats(struct net_device *net_dev,
2798 - struct ethtool_stats *stats,
2800 +static void dpaa2_eth_get_ethtool_stats(struct net_device *net_dev,
2801 + struct ethtool_stats *stats,
2804 int i; /* Current index in the data array */
2806 @@ -236,9 +224,9 @@ static void dpaa2_get_ethtool_stats(stru
2810 - u64 portal_busy = 0;
2811 + u64 portal_busy = 0, pull_err = 0;
2812 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
2813 - struct dpaa2_eth_stats *extras;
2814 + struct dpaa2_eth_drv_stats *extras;
2815 struct dpaa2_eth_ch_stats *ch_stats;
2818 @@ -266,16 +254,18 @@ static void dpaa2_get_ethtool_stats(stru
2819 ch_stats = &priv->channel[j]->stats;
2820 cdan += ch_stats->cdan;
2821 portal_busy += ch_stats->dequeue_portal_busy;
2822 + pull_err += ch_stats->pull_err;
2825 *(data + i++) = portal_busy;
2826 + *(data + i++) = pull_err;
2827 *(data + i++) = cdan;
2829 #ifdef CONFIG_FSL_QBMAN_DEBUG
2830 for (j = 0; j < priv->num_fqs; j++) {
2831 /* Print FQ instantaneous counts */
2832 err = dpaa2_io_query_fq_count(NULL, priv->fq[j].fqid,
2836 netdev_warn(net_dev, "FQ query error %d", err);
2838 @@ -303,12 +293,12 @@ static void dpaa2_get_ethtool_stats(stru
2842 -static const struct dpaa2_hash_fields {
2843 +static const struct dpaa2_eth_hash_fields {
2845 enum net_prot cls_prot;
2848 -} dpaa2_hash_fields[] = {
2849 +} hash_fields[] = {
2852 .rxnfc_field = RXH_L2DA,
2853 @@ -353,55 +343,53 @@ static const struct dpaa2_hash_fields {
2857 -static int dpaa2_cls_is_enabled(struct net_device *net_dev, u64 flag)
2858 +static int cls_is_enabled(struct net_device *net_dev, u64 flag)
2860 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
2862 return !!(priv->rx_hash_fields & flag);
2865 -static int dpaa2_cls_key_off(struct net_device *net_dev, u64 flag)
2866 +static int cls_key_off(struct net_device *net_dev, u64 flag)
2870 - for (i = 0; i < ARRAY_SIZE(dpaa2_hash_fields); i++) {
2871 - if (dpaa2_hash_fields[i].rxnfc_field & flag)
2872 + for (i = 0; i < ARRAY_SIZE(hash_fields); i++) {
2873 + if (hash_fields[i].rxnfc_field & flag)
2875 - if (dpaa2_cls_is_enabled(net_dev,
2876 - dpaa2_hash_fields[i].rxnfc_field))
2877 - off += dpaa2_hash_fields[i].size;
2878 + if (cls_is_enabled(net_dev, hash_fields[i].rxnfc_field))
2879 + off += hash_fields[i].size;
2885 -static u8 dpaa2_cls_key_size(struct net_device *net_dev)
2886 +static u8 cls_key_size(struct net_device *net_dev)
2890 - for (i = 0; i < ARRAY_SIZE(dpaa2_hash_fields); i++) {
2891 - if (!dpaa2_cls_is_enabled(net_dev,
2892 - dpaa2_hash_fields[i].rxnfc_field))
2893 + for (i = 0; i < ARRAY_SIZE(hash_fields); i++) {
2894 + if (!cls_is_enabled(net_dev, hash_fields[i].rxnfc_field))
2896 - size += dpaa2_hash_fields[i].size;
2897 + size += hash_fields[i].size;
2903 -static u8 dpaa2_cls_max_key_size(struct net_device *net_dev)
2904 +static u8 cls_max_key_size(struct net_device *net_dev)
2908 - for (i = 0; i < ARRAY_SIZE(dpaa2_hash_fields); i++)
2909 - size += dpaa2_hash_fields[i].size;
2910 + for (i = 0; i < ARRAY_SIZE(hash_fields); i++)
2911 + size += hash_fields[i].size;
2916 -void dpaa2_cls_check(struct net_device *net_dev)
2917 +void check_fs_support(struct net_device *net_dev)
2919 - u8 key_size = dpaa2_cls_max_key_size(net_dev);
2920 + u8 key_size = cls_max_key_size(net_dev);
2921 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
2923 if (priv->dpni_attrs.options & DPNI_OPT_DIST_FS &&
2924 @@ -417,7 +405,7 @@ void dpaa2_cls_check(struct net_device *
2925 /* Set RX hash options
2926 * flags is a combination of RXH_ bits
2928 -int dpaa2_set_hash(struct net_device *net_dev, u64 flags)
2929 +int dpaa2_eth_set_hash(struct net_device *net_dev, u64 flags)
2931 struct device *dev = net_dev->dev.parent;
2932 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
2933 @@ -441,11 +429,11 @@ int dpaa2_set_hash(struct net_device *ne
2935 memset(&cls_cfg, 0, sizeof(cls_cfg));
2937 - for (i = 0; i < ARRAY_SIZE(dpaa2_hash_fields); i++) {
2938 + for (i = 0; i < ARRAY_SIZE(hash_fields); i++) {
2939 struct dpkg_extract *key =
2940 &cls_cfg.extracts[cls_cfg.num_extracts];
2942 - if (!(flags & dpaa2_hash_fields[i].rxnfc_field))
2943 + if (!(flags & hash_fields[i].rxnfc_field))
2946 if (cls_cfg.num_extracts >= DPKG_MAX_NUM_OF_EXTRACTS) {
2947 @@ -454,14 +442,12 @@ int dpaa2_set_hash(struct net_device *ne
2950 key->type = DPKG_EXTRACT_FROM_HDR;
2951 - key->extract.from_hdr.prot =
2952 - dpaa2_hash_fields[i].cls_prot;
2953 + key->extract.from_hdr.prot = hash_fields[i].cls_prot;
2954 key->extract.from_hdr.type = DPKG_FULL_FIELD;
2955 - key->extract.from_hdr.field =
2956 - dpaa2_hash_fields[i].cls_field;
2957 + key->extract.from_hdr.field = hash_fields[i].cls_field;
2958 cls_cfg.num_extracts++;
2960 - enabled_flags |= dpaa2_hash_fields[i].rxnfc_field;
2961 + enabled_flags |= hash_fields[i].rxnfc_field;
2964 dma_mem = kzalloc(DPAA2_CLASSIFIER_DMA_SIZE, GFP_DMA | GFP_KERNEL);
2965 @@ -486,7 +472,7 @@ int dpaa2_set_hash(struct net_device *ne
2969 - dist_cfg.dist_size = dpaa2_queue_count(priv);
2970 + dist_cfg.dist_size = dpaa2_eth_queue_count(priv);
2971 if (dpaa2_eth_fs_enabled(priv)) {
2972 dist_cfg.dist_mode = DPNI_DIST_MODE_FS;
2973 dist_cfg.fs_cfg.miss_action = DPNI_FS_MISS_HASH;
2974 @@ -508,14 +494,14 @@ int dpaa2_set_hash(struct net_device *ne
2978 -static int dpaa2_cls_prep_rule(struct net_device *net_dev,
2979 - struct ethtool_rx_flow_spec *fs,
2981 +static int prep_cls_rule(struct net_device *net_dev,
2982 + struct ethtool_rx_flow_spec *fs,
2985 struct ethtool_tcpip4_spec *l4ip4_h, *l4ip4_m;
2986 struct ethhdr *eth_h, *eth_m;
2987 struct ethtool_flow_ext *ext_h, *ext_m;
2988 - const u8 key_size = dpaa2_cls_key_size(net_dev);
2989 + const u8 key_size = cls_key_size(net_dev);
2990 void *msk = key + key_size;
2992 memset(key, 0, key_size * 2);
2993 @@ -546,51 +532,47 @@ l4ip4:
2994 "ToS is not supported for IPv4 L4\n");
2997 - if (l4ip4_m->ip4src &&
2998 - !dpaa2_cls_is_enabled(net_dev, RXH_IP_SRC)) {
2999 + if (l4ip4_m->ip4src && !cls_is_enabled(net_dev, RXH_IP_SRC)) {
3000 netdev_err(net_dev, "IP SRC not supported!\n");
3003 - if (l4ip4_m->ip4dst &&
3004 - !dpaa2_cls_is_enabled(net_dev, RXH_IP_DST)) {
3005 + if (l4ip4_m->ip4dst && !cls_is_enabled(net_dev, RXH_IP_DST)) {
3006 netdev_err(net_dev, "IP DST not supported!\n");
3009 - if (l4ip4_m->psrc &&
3010 - !dpaa2_cls_is_enabled(net_dev, RXH_L4_B_0_1)) {
3011 + if (l4ip4_m->psrc && !cls_is_enabled(net_dev, RXH_L4_B_0_1)) {
3012 netdev_err(net_dev, "PSRC not supported, ignored\n");
3015 - if (l4ip4_m->pdst &&
3016 - !dpaa2_cls_is_enabled(net_dev, RXH_L4_B_2_3)) {
3017 + if (l4ip4_m->pdst && !cls_is_enabled(net_dev, RXH_L4_B_2_3)) {
3018 netdev_err(net_dev, "PDST not supported, ignored\n");
3022 - if (dpaa2_cls_is_enabled(net_dev, RXH_IP_SRC)) {
3023 - *(u32 *)(key + dpaa2_cls_key_off(net_dev, RXH_IP_SRC))
3024 + if (cls_is_enabled(net_dev, RXH_IP_SRC)) {
3025 + *(u32 *)(key + cls_key_off(net_dev, RXH_IP_SRC))
3027 - *(u32 *)(msk + dpaa2_cls_key_off(net_dev, RXH_IP_SRC))
3028 + *(u32 *)(msk + cls_key_off(net_dev, RXH_IP_SRC))
3031 - if (dpaa2_cls_is_enabled(net_dev, RXH_IP_DST)) {
3032 - *(u32 *)(key + dpaa2_cls_key_off(net_dev, RXH_IP_DST))
3033 + if (cls_is_enabled(net_dev, RXH_IP_DST)) {
3034 + *(u32 *)(key + cls_key_off(net_dev, RXH_IP_DST))
3036 - *(u32 *)(msk + dpaa2_cls_key_off(net_dev, RXH_IP_DST))
3037 + *(u32 *)(msk + cls_key_off(net_dev, RXH_IP_DST))
3041 - if (dpaa2_cls_is_enabled(net_dev, RXH_L4_B_0_1)) {
3042 - *(u32 *)(key + dpaa2_cls_key_off(net_dev, RXH_L4_B_0_1))
3043 + if (cls_is_enabled(net_dev, RXH_L4_B_0_1)) {
3044 + *(u32 *)(key + cls_key_off(net_dev, RXH_L4_B_0_1))
3046 - *(u32 *)(msk + dpaa2_cls_key_off(net_dev, RXH_L4_B_0_1))
3047 + *(u32 *)(msk + cls_key_off(net_dev, RXH_L4_B_0_1))
3051 - if (dpaa2_cls_is_enabled(net_dev, RXH_L4_B_2_3)) {
3052 - *(u32 *)(key + dpaa2_cls_key_off(net_dev, RXH_L4_B_2_3))
3053 + if (cls_is_enabled(net_dev, RXH_L4_B_2_3)) {
3054 + *(u32 *)(key + cls_key_off(net_dev, RXH_L4_B_2_3))
3056 - *(u32 *)(msk + dpaa2_cls_key_off(net_dev, RXH_L4_B_2_3))
3057 + *(u32 *)(msk + cls_key_off(net_dev, RXH_L4_B_2_3))
3061 @@ -609,12 +591,10 @@ l4ip4:
3065 - if (dpaa2_cls_is_enabled(net_dev, RXH_L2DA)) {
3066 - ether_addr_copy(key
3067 - + dpaa2_cls_key_off(net_dev, RXH_L2DA),
3068 + if (cls_is_enabled(net_dev, RXH_L2DA)) {
3069 + ether_addr_copy(key + cls_key_off(net_dev, RXH_L2DA),
3071 - ether_addr_copy(msk
3072 - + dpaa2_cls_key_off(net_dev, RXH_L2DA),
3073 + ether_addr_copy(msk + cls_key_off(net_dev, RXH_L2DA),
3076 if (!is_zero_ether_addr(eth_m->h_dest)) {
3077 @@ -639,12 +619,10 @@ l4ip4:
3081 - if (dpaa2_cls_is_enabled(net_dev, RXH_L2DA)) {
3082 - ether_addr_copy(key
3083 - + dpaa2_cls_key_off(net_dev, RXH_L2DA),
3084 + if (cls_is_enabled(net_dev, RXH_L2DA)) {
3085 + ether_addr_copy(key + cls_key_off(net_dev, RXH_L2DA),
3087 - ether_addr_copy(msk
3088 - + dpaa2_cls_key_off(net_dev, RXH_L2DA),
3089 + ether_addr_copy(msk + cls_key_off(net_dev, RXH_L2DA),
3092 if (!is_zero_ether_addr(ext_m->h_dest)) {
3093 @@ -657,9 +635,9 @@ l4ip4:
3097 -static int dpaa2_do_cls(struct net_device *net_dev,
3098 - struct ethtool_rx_flow_spec *fs,
3100 +static int do_cls(struct net_device *net_dev,
3101 + struct ethtool_rx_flow_spec *fs,
3104 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
3105 const int rule_cnt = DPAA2_CLASSIFIER_ENTRY_COUNT;
3106 @@ -674,19 +652,19 @@ static int dpaa2_do_cls(struct net_devic
3109 if ((fs->ring_cookie != RX_CLS_FLOW_DISC &&
3110 - fs->ring_cookie >= dpaa2_queue_count(priv)) ||
3111 + fs->ring_cookie >= dpaa2_eth_queue_count(priv)) ||
3112 fs->location >= rule_cnt)
3115 memset(&rule_cfg, 0, sizeof(rule_cfg));
3116 - rule_cfg.key_size = dpaa2_cls_key_size(net_dev);
3117 + rule_cfg.key_size = cls_key_size(net_dev);
3119 /* allocate twice the key size, for the actual key and for mask */
3120 dma_mem = kzalloc(rule_cfg.key_size * 2, GFP_DMA | GFP_KERNEL);
3124 - err = dpaa2_cls_prep_rule(net_dev, fs, dma_mem);
3125 + err = prep_cls_rule(net_dev, fs, dma_mem);
3129 @@ -735,13 +713,13 @@ err_free_mem:
3133 -static int dpaa2_add_cls(struct net_device *net_dev,
3134 - struct ethtool_rx_flow_spec *fs)
3135 +static int add_cls(struct net_device *net_dev,
3136 + struct ethtool_rx_flow_spec *fs)
3138 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
3141 - err = dpaa2_do_cls(net_dev, fs, true);
3142 + err = do_cls(net_dev, fs, true);
3146 @@ -751,12 +729,12 @@ static int dpaa2_add_cls(struct net_devi
3150 -static int dpaa2_del_cls(struct net_device *net_dev, int location)
3151 +static int del_cls(struct net_device *net_dev, int location)
3153 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
3156 - err = dpaa2_do_cls(net_dev, &priv->cls_rule[location].fs, false);
3157 + err = do_cls(net_dev, &priv->cls_rule[location].fs, false);
3161 @@ -765,7 +743,7 @@ static int dpaa2_del_cls(struct net_devi
3165 -static void dpaa2_clear_cls(struct net_device *net_dev)
3166 +static void clear_cls(struct net_device *net_dev)
3168 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
3170 @@ -774,7 +752,7 @@ static void dpaa2_clear_cls(struct net_d
3171 if (!priv->cls_rule[i].in_use)
3174 - err = dpaa2_del_cls(net_dev, i);
3175 + err = del_cls(net_dev, i);
3177 netdev_warn(net_dev,
3178 "err trying to delete classification entry %d\n",
3179 @@ -782,8 +760,8 @@ static void dpaa2_clear_cls(struct net_d
3183 -static int dpaa2_set_rxnfc(struct net_device *net_dev,
3184 - struct ethtool_rxnfc *rxnfc)
3185 +static int dpaa2_eth_set_rxnfc(struct net_device *net_dev,
3186 + struct ethtool_rxnfc *rxnfc)
3190 @@ -792,19 +770,19 @@ static int dpaa2_set_rxnfc(struct net_de
3191 /* first off clear ALL classification rules, chaging key
3192 * composition will break them anyway
3194 - dpaa2_clear_cls(net_dev);
3195 + clear_cls(net_dev);
3196 /* we purposely ignore cmd->flow_type for now, because the
3197 * classifier only supports a single set of fields for all
3200 - err = dpaa2_set_hash(net_dev, rxnfc->data);
3201 + err = dpaa2_eth_set_hash(net_dev, rxnfc->data);
3203 case ETHTOOL_SRXCLSRLINS:
3204 - err = dpaa2_add_cls(net_dev, &rxnfc->fs);
3205 + err = add_cls(net_dev, &rxnfc->fs);
3208 case ETHTOOL_SRXCLSRLDEL:
3209 - err = dpaa2_del_cls(net_dev, rxnfc->fs.location);
3210 + err = del_cls(net_dev, rxnfc->fs.location);
3214 @@ -814,8 +792,8 @@ static int dpaa2_set_rxnfc(struct net_de
3218 -static int dpaa2_get_rxnfc(struct net_device *net_dev,
3219 - struct ethtool_rxnfc *rxnfc, u32 *rule_locs)
3220 +static int dpaa2_eth_get_rxnfc(struct net_device *net_dev,
3221 + struct ethtool_rxnfc *rxnfc, u32 *rule_locs)
3223 struct dpaa2_eth_priv *priv = netdev_priv(net_dev);
3224 const int rule_cnt = DPAA2_CLASSIFIER_ENTRY_COUNT;
3225 @@ -831,7 +809,7 @@ static int dpaa2_get_rxnfc(struct net_de
3228 case ETHTOOL_GRXRINGS:
3229 - rxnfc->data = dpaa2_queue_count(priv);
3230 + rxnfc->data = dpaa2_eth_queue_count(priv);
3233 case ETHTOOL_GRXCLSRLCNT:
3234 @@ -868,15 +846,13 @@ static int dpaa2_get_rxnfc(struct net_de
3237 const struct ethtool_ops dpaa2_ethtool_ops = {
3238 - .get_drvinfo = dpaa2_get_drvinfo,
3239 - .get_msglevel = dpaa2_get_msglevel,
3240 - .set_msglevel = dpaa2_set_msglevel,
3241 + .get_drvinfo = dpaa2_eth_get_drvinfo,
3242 .get_link = ethtool_op_get_link,
3243 - .get_settings = dpaa2_get_settings,
3244 - .set_settings = dpaa2_set_settings,
3245 - .get_sset_count = dpaa2_get_sset_count,
3246 - .get_ethtool_stats = dpaa2_get_ethtool_stats,
3247 - .get_strings = dpaa2_get_strings,
3248 - .get_rxnfc = dpaa2_get_rxnfc,
3249 - .set_rxnfc = dpaa2_set_rxnfc,
3250 + .get_settings = dpaa2_eth_get_settings,
3251 + .set_settings = dpaa2_eth_set_settings,
3252 + .get_sset_count = dpaa2_eth_get_sset_count,
3253 + .get_ethtool_stats = dpaa2_eth_get_ethtool_stats,
3254 + .get_strings = dpaa2_eth_get_strings,
3255 + .get_rxnfc = dpaa2_eth_get_rxnfc,
3256 + .set_rxnfc = dpaa2_eth_set_rxnfc,